Directory Extensions for namespace listing

[vyzo]

Adds support for namespace listing to the directory protocol; see #71

Protocol changes:

• RegisterPeer is extended to include a PublisherInfo object, which includes the peer's publisher id and namespaces in its stmt db
• ListPeers is extended to include a namespace
• Adds a /mediachain/dir/listns protocol for listing namespaces from registered peers

/mediachain/dir/list extensions:

• mcnode registerImpl includes publisher info, with namespaces computed with a query.
• mcdir stores publisher info along with the peer info and supports listing with namespace filters
• API endpoint: /dir/list/{namespace}
• API endpoint: /dir/list/{namespace}/all to also include self in resluts

/mediachain/dir/listns implementation:

• directory iterates through all peers and collects set of known namespaces
• API endpoint: /dir/listns

Some (incidental) net refactoring:

• Refector common directory connection logic to dirConnect
• Refactor doConnect to also open a stream, similar to doDirConnect

Example:

# test network
$ mcdir -d /tmp/mcdir
...
I am /ip4/127.0.0.1/tcp/9000/p2p/QmSg7QwnV5xPEy7zjFx7zEMYumJHwYWPyY1RggTYqtUrif
...

$ mcnode -d /tmp/mcnode1
...
Peer ID: QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r
...

$ mcnode -d /tmp/mcnode2 -l 9003 -c 9004
...
Peer ID: QmPsrVV6mGkha31VXq9HggsSs1iKX6D7hnii1CacQUSCK3
...

# seed the nodes with some data
$ mcclient -p http://localhost:9002 status online
status set to online
$ mcclient -p http://localhost:9002 merge QmeiY2eHMwK92Zt6X4kUUC3MsjMmVb2VnGZ17DhnhRPCEQ "SELECT * FROM * LIMIT 10"
merged 10 statements and 10 objects
$ mcclient -p http://localhost:9002 merge QmZ6dckUhRouVr6AsBTpK6vMLVpcz1KAeJAJVQEZQ5gCek "SELECT * FROM * LIMIT 10"
merged 10 statements and 900 objects
$ mcclient -p http://localhost:9002 query "SELECT namespace FROM *"
"images.dpla"
"images.flickr"
"mediachain.schemas"

$ mcclient -p http://localhost:9004 status online
status set to online
$ mcclient -p http://localhost:9004 merge QmTVsocFCSEdPyM8dZ734GRhjvvmYyL9fShyezVkbPj17E "SELECT * FROM * LIMIT 10"
merged 10 statements and 10 objects
$ mcclient -p http://localhost:9004 query "SELECT namespace FROM *"
"mediachain.schemas"
"museums.moma.artists"

# go public with the test directory
$ mcclient -p http://localhost:9002 config dir /ip4/127.0.0.1/tcp/9000/p2p/QmSg7QwnV5xPEy7zjFx7zEMYumJHwYWPyY1RggTYqtUrif
$ mcclient -p http://localhost:9002 status public

$ mcclient -p http://localhost:9004 config dir /ip4/127.0.0.1/tcp/9000/p2p/QmSg7QwnV5xPEy7zjFx7zEMYumJHwYWPyY1RggTYqtUrif
$ mcclient -p http://localhost:9004 status public
status set to public

# view from each node
$ curl http://localhost:9002/dir/list
QmPsrVV6mGkha31VXq9HggsSs1iKX6D7hnii1CacQUSCK3
$ curl http://localhost:9004/dir/list
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r

$ curl http://localhost:9002/dir/listns
images.dpla
images.flickr
mediachain.schemas
museums.moma.artists

$ curl http://localhost:9002/dir/list/images.dpla
$ curl http://localhost:9004/dir/list/images.dpla
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r
$ curl http://localhost:9002/dir/list/images.flickr
$ curl http://localhost:9004/dir/list/images.flickr
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r
$ curl http://localhost:9002/dir/list/mediachain.schemas
QmPsrVV6mGkha31VXq9HggsSs1iKX6D7hnii1CacQUSCK3
$ curl http://localhost:9004/dir/list/mediachain.schemas
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r
$ curl http://localhost:9002/dir/list/museums.moma.artists
QmPsrVV6mGkha31VXq9HggsSs1iKX6D7hnii1CacQUSCK3
$ curl http://localhost:9004/dir/list/museums.moma.artists
$ curl "http://localhost:9002/dir/list/images.*"
$ curl "http://localhost:9004/dir/list/images.*"
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r

# include self in results
$ curl "http://localhost:9002/dir/list/*/all"
QmPsrVV6mGkha31VXq9HggsSs1iKX6D7hnii1CacQUSCK3
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r
$ curl "http://localhost:9002/dir/list/images.*/all"
QmaiJkP7eNQ7HSBWR8KF5qnWDenwiFa8X6kd4heMrCE16r

[yusefnapora]

This looks great 👍

Newly added namespaces should get picked up by the directory on the next register "heartbeat", right? So ~5mins is the max time between adding a namespace and it being listed in the directory.

I wonder if we should add some way for a peer to see its own listings from the directory; I notice that your own peer's namespaces are excluded by default, which seems correct. But it might be good for a user to be able to verify that their stuff is visible to other peers without having to spin up another testing node...

Also (only semi-related), can we use the DHT provider records to distribute namespace listings without a directory? That might be cool, but seems like a distributed cache invalidation problem we might want to avoid 😄

[vyzo]

Newly added namespaces should get picked up by the directory on the next register "heartbeat", right?

yes

I wonder if we should add some way for a peer to see its own listings from the directory

The filtering happens in the client side, ie the concat node simply edits itself out of the list. We could change this, but it complicates some usage to have your own self mixed up with your peers.

Also (only semi-related), can we use the DHT provider records to distribute namespace listings without a directory?

It's part of the grand plan :)

[vyzo]

Added the /dir/list/{ns}/all variant to include self in results, which lets the user choose whether he wants to include self in results and avoids the impedance mismatch.