Provide an example of multi-device setup in a README.md

[samo3l]

I am really sorry about that. But, I spent a lot of time trying to setup multi-hyperbee on 2 different devices and write/read some simple data from it. And looks like it's impossible to do without a good knowledge of node js. It will be very helpful for peoples who are not well experienced with node.js to have an example of such a configuration. Could you please share it with us !? Sorry again for a lot of questions and many many thanks!

[samo3l]

@pgmemk could you please take a look. I am trying to use such a script to do that.

NODE1:

const { Client } = require('hyperspace')
const Hyperbee = require('hyperbee')
const MultiHyperbee = require('multi-hyperbee')
const hypercore = require('hypercore')
const hyperswarm = require('hyperswarm')
const crypto = require('crypto')
const multifeed = require('multifeed')
const pump = require('pump')

const hyperbeeOpts = { keyEncoding: 'utf-8', valueEncoding: 'json' }

const topicHex = crypto.createHash('sha256')
  .update('imdb')
  .digest()

const db = new MultiHyperbee('db1', hyperbeeOpts )
db.addPeer('98br6bd758c7124d2e6552457sda34704ac490ec83eb7c1876dfqe32f50c3aeb') // 2nd VM diff key (also tried db.key)

start()

let data = {
  object0: {
    firstName: 'J',
    lastName: 'S',
    friends: ['Claire', 'Martha', 'Jake', 'Sean']
  }
}

async function start() {
  await db.ready()
  await startSwarm(topicHex)
  await db.put('Contact/r1', data)
}

async function startSwarm(topic) {
  var swarm = hyperswarm()
  swarm.join(topic, {
    lookup: true,
    announce: true
  })
  const diffFeed = (await db.getDiff()).feed
  console.log(diffFeed.key.toString('hex'))
  swarm.on('connection', function(connection, info) {
    console.log('(New peer connected!)')
    pump(diffFeed.replicate(true, {live: true}), db.feed.replicate(true, { live: true }), diffFeed.replicate(true, {live: true}))
    console.log(db)
    logFeed(db.feed)
  })
}

function logFeed(feed) {
  console.log('watching', feed.key.toString('hex'), feed.length)
  feed.createReadStream({ live: true })
    .on('data', function(data) {
      console.log(data.toString('ascii'))
  })
}

NODE2:

const { Client } = require('hyperspace')
const Hyperbee = require('hyperbee')
const MultiHyperbee = require('multi-hyperbee')
const hypercore = require('hypercore')
const hyperswarm = require('hyperswarm')
const crypto = require('crypto')
const multifeed = require('multifeed')
const pump = require('pump')

const hyperbeeOpts = { keyEncoding: 'utf-8', valueEncoding: 'json' }

const topicHex = crypto.createHash('sha256')
  .update('imdb')
  .digest()

const db = new MultiHyperbee('db1', hyperbeeOpts )
db.addPeer('7403b4d3901d10385456be896afcf41a7acb4b6f2fb7b3bdf4f103344efaa6b1') // 1st diff key (also tried db.key)

start()

async function start() {
  await db.ready()
  await startSwarm(topicHex)
}

async function startSwarm(topic) {
  var swarm = hyperswarm()
  swarm.join(topic, {
    lookup: true,
    announce: true
  })
  const diffFeed = (await db.getDiff()).feed
  console.log(diffFeed.key.toString('hex'))
  swarm.on('connection', function(connection, info) {
    console.log('(New peer connected!)')
    pump(diffFeed.replicate(false, {live: true}), db.feed.replicate(false, { live: true }), diffFeed.replicate(false, {live: true}))
    console.log(db)
    logFeed(db.feed)
  })
}

function logFeed(feed) {
  console.log('watching', feed.key.toString('hex'), feed.length)
  feed.createReadStream({ live: true })
    .on('data', function(data) {
      console.log(data.toString('ascii'))
  })
}

Expected result - I should see the same output on both nodes:

 {
    firstName: 'J',
    lastName: 'S',
    friends: ['Claire', 'Martha', 'Jake', 'Sean']
  }

But I can see that only on the 1st node. 2nd node is not able to access data.

[pgmemk]

I will look more, but I see couple of bugs here:

• db.addPeer(...) is async so you need to await for it.
• same with start()

• The snippet of the code below is not correct.

  • You have true in all replicate calls which means that all of the streams are the initiators of the connection.
  • Another thing you are pumping the diffFeed to store feed db.feed. Instead you should pump the diffFeed to connection you received

    
    swarm.on('connection', function(connection, info) {
    console.log('(New peer connected!)')
    // this is wrong
    pump(diffFeed.replicate(true, {live: true}), db.feed.replicate(true, { live: true }), diffFeed.replicate(true, {live: true}))      
    // this is better :)
    let dstream = diffFeed.replicate(info.client, {live: true})
    pump(dstream, connection, dstream)

  console.log(db) logFeed(db.feed) })

[samo3l]

Many thanks again. Applied all your suggestions but still not able to get data from another device. I also see that all tests run locally, did you tried to use it on 2 different devices, for example, Linux VMs?

I was able to read data from another device using native hypercore. Single writer hyperbee also works for me. I was able to run some simple Linux terminal P2P chat, based on multifeed with multiple writers. And wanna do something similar based on multihyperbee. Will be very thankful one more time for any input.

[samo3l]

@pgmemk let me know if I can be useful.

[pgmemk]

@samostrovskyi Thank you for you feedback. Please check the example here. I tried for 2 processes. You can try for more and let me know if it works.

This is still work in progress which means it is bound to improve :).

[pgmemk]

@samostrovskyi I made more changes 😊. Please make sure to use the latest version.

[urbien]

@samostrovskyi last change makes API more compliant with hyperbee, take a look at the readme for usage example and a link to example code that was tested to work. You should be on your way to success now :-) Apologies for the bump on the road.

[samo3l]

@urbien @pgmemk, I am very thankful for your efforts in such a short period of time. It works perfectly now on 2 remote Ubuntu VMs yet. But 3d one can't replicate data. It's not urgent at least for me now, and I am sure it's possible to fix and improve. Many thanks one more time. Ping me If I might be useful somewhere.

[pgmemk]

@samostrovskyi Thank you again. You are helping by finding these glitches. I fixed the problem and the doc on how to run the example here.

Just pass the keys separated with comma like this

node examples/example.js -s [key1 storage name] -k key2,key3

[pgmemk]

@samostrovskyi You offered to help. That would be awesome!!! We would really appreciate it. If you still feel this way here are couple of ideas, choose any you are comfortable with. I'll help as much as my understanding allows 😀 and @urbien might help as well.

• Create more tests for CRDT merging. With CRDT you get the same hyperbee state on all peers. CRDT stands for Conflict-free Replicated Data Type. With CRDT concurrent modifications are merged even if they are in different order.
• You can also think about the issue I posted here
• You can also think of what else to think 🤔 😁 Thank you

[pgmemk]

the issue was resolved