Rendezvous discovery service

[vasco-santos]

Overview

Libp2p rendezvous is lightweight mechanism for peer discovery that is more flexible and configurable than other discovery protocols libp2p currently supports.

Aiming to enable this protocol maximum flexibility, it provides a programatic API for register, unregister and discover. The same as the WIP go implementation and exactly matches the flows specified in the spec.

libp2p users in JS land are used to just insert a discovery service within a config and will work work out of the box, not only for discovering the peer, but also to add its addresses to the AddressBook, to emit a peer:discovery event and also to eventually dial it, if the autoDial option is enabled.

While this proposal might change some stuff around the current libp2p integration proposed in libp2p/js-libp2p-rendezvous/blob/feat/rendezvous-protocol-full-implementation/LIBP2P.md, it is important to read it before reading this proposal.

Proposal

Taking into account that libp2p rendezvous is a discovery protocol, it should work out of the box as any other peer discovery discovery in JS, that is, have exactly the same flows.

Discovery Service API

The rendezvous implementation should include a discovery service in the same fashion as the webrtc-star transport has:

const rendezvous = new Rendezvous()

rendezvous.discovery.on('peer', (peer) => {

))
rendezvous.discovery.start()
rendezvous.discovery.stop()

With this discovery service, libp2p would be able to integrate rendezvous as all the other discovery services and all the expected flows will be the same. It should receive a similar configuration as the other discovery services:

{
  discovery: {
    enabled: true,
    interval: 1000
  }
}

interface peer-discovery

The current implementation of this interface emits an event id PeerId and an array of multiaddrs. In the rendezvous context, as well as future contexts, like DHT random walk discovery and even Gossipsub peer exchange discovery, the peers will exchange signed peer records. The interface peer-discovery peer event should be extended to support them:

discovery.on('peer', ({ id, multiaddrs, signedPeerRecord } => {})

It might be nice to also store the tag of the discovery service responsible for discovering an address in the AddressBook.

Rendezvous configuration

To support a full autonomous rendezvous discovery, this module needs to receive in advance a set of namespaces that the node should register on and discover other peers in.

{
  enabled: true,
  namespaces: ['/namespace/1', `/namespace/2`]
}

These namespaces should be registered in start up and refreshed over time to avoid ttl removals by the server garbage collector. Moreover, these namespaces will be used by the discovery service to discover other peers within the configured interval.

Power Users

While most users will expect to use rendezvous as a out of the box piece, others might want to implement more sophisticated flows. Examples are unregistering from a namespace when we have enough circuit relay nodes connected, or register to a given namespace if the number of connected peers is less than a certain level.

These type of use cases should be achieved through the application layer and libp2p should enable them. As a result, libp2p should expose the rendezvous API (register, unregister and discover).

In register, the user should be able to provide options for keeping the registration refreshed over time and to have the discovery service to automatically look for peers. However, it should also allow a single registration with no discover automation.

unregister should remove the given namespace from the list of namespaces to refresh registrations and discover.

Users should be able to do their own discover queries by a given namespace. It is important to mention, that in this case the typical discovery service features like peer:discover event and information added to the AddressBook will not be provided by libp2p out of the box.

Rendezvous in libp2p config

We might have one big difference from other discovery services. We might move the rendezvous implementation inside libp2p core, which will have impact on how we enable and configure the discovery service compared to other modules.

We currently provide the module and its configuration through config as:

const Libp2p = require('libp2p')
const TCP = require('libp2p-tcp')
const MPLEX = require('libp2p-mplex')
const SECIO = require('libp2p-secio')
const MulticastDNS = require('libp2p-mdns')

const node = await Libp2p.create({
  modules: {
    transport: [TCP],
    streamMuxer: [MPLEX],
    connEncryption: [SECIO],
    peerDiscovery: [MulticastDNS]
  },
  config: {
    peerDiscovery: {
      [MulticastDNS.tag]: {
        interval: 1000,
        enabled: true
      }
    }
  }
})

If we move rendezvous to core libp2p, we need to find a clean alternative for this. I was inclined to move this to core, but I have been shifting my idea over time.

[vasco-santos]

@jacobheun can you let me know your thoughts on this?

[vasco-santos]

Regarding the last point (Rendezvous in libp2p config), I have thought about this a bit. What if we have the rendezvous protocol implementation in core, with its own API exposed as libp2p.rendezvous and all the options mentioned above except for the discovery ones.

Then, we could have a libp2p-rendezvous-discovery module, which would be pluggable in the discovery modules via config. Discovery modules receive libp2p, and this service would be responsible for filling the gap for automatic discovery with continuous discovery query with events and stuff added to the addressBook.

[jacobheun]

I'm hesitant to bring this into core right now. Having it external would allow us more time to evaluate the API here and see if it makes sense to pull this into core later. There are a lot of considerations to keep in mind in terms of registration TTL's, reregistration, discovery rate, unregistration, etc, and that is going to depend on the type of node and possibly the specific namespace. Baking this all into configuration sounds convoluted to me.

I think this is really where functional configuration of libp2p could make this nicer. Then all the setup and "rules" for rendezvous could be handled inside that single function. We could work to time these for the same release of libp2p (0.30?).

[vasco-santos]

I think this is really where functional configuration of libp2p could make this nicer. Then all the setup and "rules" for rendezvous could be handled inside that single function. We could work to time these for the same release of libp2p (0.30?).

Yes! I totally agree with you. Let's start by having it outside libp2p core and also have feedback from the community!

The remaining question is how libp2p would integrate rendezvous and its discovery. I think we can start by having it together in the same module, and eventually split the discovery when we move this to core. This way, we can simply do:

const Rendezvous = require('libp2p-rendezvous')

const node = await Libp2p.create()
const rendezvous = new Rendezvous({ libp2p }) // Set other options

await node.start()
await rendezvous.start()

This would have the discovery integrated out of the box and users could still use the rendezvous API for more control, if they want to.

[jacobheun]

I think we can start by having it together in the same module, and eventually split the discovery when we move this to core.

I don't know that they'll ever need to be separate unless for some reason you didn't want libp2p to automatically become aware of the peers which seems odd. So yes, I'd keep them together and we can see later if this is warranted.

[ROBERT-MCDOWELL]

any news of this very nice feature?