P2P overlay networks.

[klueq]

Can DAT be used as a library to create an overlay network of nodes interested in topic foo/1.2.3 and let them exchange with arbitrary messages? Number of nodes: ~1M. Message size: < 1 KB.

For example, I want to make a distributed p2p version of a books catalogue. Ther catalogue has a well-known name, e.g. books/1.2.3. Nodes use this name discover other nodes connected to this overlay network. Once connected, they download the list of book ids from a peer. If someone wants to add a book, they send its id to a few peers, they add it to the local storage and re-transmit to other peers. An important property of this system is that all the nodes are equal: there is no admin who has special rights.

API needed for this:

• join("books/1.2.3") finds a few peers in this overlay network
• peers("books/1.2.3") returns the list of online peers we're currently connected to
• send("books/1.2.3", peerId, "howdy") sends a message to the given peer
• listen("books/1.2.3", callback) gets messages sent by other peers with the send() API

Thoughts?

[KrishnaPG]

You can use two hypercores. One for discovery and the other for the actual storage.

They have one rough example on the ReadMe

// Create a hypercore
// Check out the hypercore docs for what you can do with it
// https://github.com/mafintosh/hypercore
const myCore = Hypercore(null, {
  valueEncoding: 'json',
  persist: false,
  // storage can be set to an instance of `random-access-*`
  // const RAI = require('random-access-idb')
  // otherwise it defaults to `random-access-web` in the browser
  // and `random-access-file` in node
  storage: null  // storage: RAI
})

// Add some data to it
myCore.append(JSON.stringify({
  name: 'Alice'
}), () => {
  // Use extension messages for sending extra data over the p2p connection
  const discoveryCoreKey = 'dat://bee80ff3a4ee5e727dc44197cb9d25bf8f19d50b0f3ad2984cfe5b7d14e75de7'
  const discoveryCore = new Hypercore(discoveryCoreKey, {
    extensions: ['discovery']
  })

  // When you find a new peer, tell them about your core
  discoveryCore.on('peer-add', (peer) => {
    console.log('Got a peer!')
    peer.extension('discovery', myCore.key)
  })

  // When a peer tells you about their core, load it
  discoveryCore.on('extension', (type, message) => {
    console.log('Got extension message', type, message)
    if (type !== 'discovery') return
    discoveryCore.close()

    const otherCore = new Hypercore(message, {
      valueEncoding: 'json',
      persist: false
    })

    // Render the peer's data from their core
    otherCore.get(0, console.log)
  })
})

This gitter im chat is a good place where people are friendly are willing to help more on this. Got this above info from there

[pfrazee]

I'm not informed enough on the details Hyperswarm to give a good response on creating a custom overlay network, but I do want to explain a bit about what @KrishnaPG's solution is doing.

Hypercores use Hyperswarm to create DHT topics. The DHT topic is used to announce your machine and find announced peers. The hypercore uses a "discovery key" as the topic; the discovery key is the hash of the core's public key. (The hash is used to hide the public key, so you have to get the public key from some other channel than the DHT to join the topic.)

Hypercore's wire protocol includes a mechanism for "extension" messages which enable custom payloads to be sent over a connection which was created via the DHT topic. This is what @KrishnaPG is leveraging. Essentially, they're piggy-backing off of Hypercore's exchange process. It does work, but it's got more overhead than may be strictly necessary. The "Discovery Hypercore" doesn't really add anything to the solution; it's just being leveraged to tap into the Hyperswarm behaviors indirectly.