Lack of documentation on replicating over network

[tdjsnelling]

I am trying to replicate a hyperdb between 1 or more peers over the internet.

Using hyperdiscovery I am able to successfully find and connect to peers, but I am stuck at actually replicating my hyperdb to them.

Is anybody able to point me in the right direction? Current code is below.

const db = hyperdb(
  path.resolve(appDir, 'keys.db'),
  process.env.DB_KEY,
  { valueEncoding: 'json' }
)

db.on('ready', () => {
  const swarm = hyperdiscovery(db, { live: true })
  console.log('database ready')

  swarm.on('connection', (peer, type) => {
    console.log(
      `a peer at ${type.host} connected. ${swarm.connections.length} total`
    )

    const stream = db.replicate()
    pump(peer, stream, peer, err => {
      if (err) {
        console.log(`pump: ${err}`)
      }
    })

    peer.on('close', () => {
      console.log(`a peer at ${type.host} disconnected`)
    })
  })
})

[joehand]

Hmmm. Using hyperdiscovery should automatically replicate, it should work without the pump bit:

var hyperdb = require('hyperdb')
var swarm = require('hyperdiscovery')

var db = hyperdb('/feed', 'ARCHIVE_KEY')
db.on('ready', function() {
  var sw = swarm(db)
})

You may also try replicating locally, to ensure that works first:

const db = hyperdb(
  path.resolve(appDir, 'keys.db'),
  process.env.DB_KEY,
  { valueEncoding: 'json' }
)

db.on('ready', () => {
  const clone = hyperdb(require('random-access-memory'), db.key) 

  const stream = db.replicate()
  pump(stream, clone.replicate(), stream, err => {
    if (err) console.error(err)
  })

  clone.get('/', (err, val) => {
    console.log(err, val)
  })
})

[tdjsnelling]

My mistake, I took a deeper look at hyperdiscovery after I posted this and realised that it should handle all of the replication.

However, I still can't see any replication after removing my pump code. Local replication does not work either as per your example, I see zero output. No errors.

[joehand]

Hmm, does it work with fresh dbs in memory? What version are you on for hyperdb/node/etc. Running with DEBUG=* node file.js may give you more info.

The basic example would be:

const ram = require('random-access-memory')
const hyperdb = require('hyperdb')
const pump = require('pump')

const db = hyperdb(ram)

db.put('/hello', 'world',  (err) => {})

db.on('ready', () => {
  const clone = hyperdb(ram, db.key) 

  const stream = db.replicate()
  pump(stream, clone.replicate(), stream, err => {
    if (err) console.error(err)
  })

  clone.get('/hello', (err, nodes) => {
    if (err) throw err
    console.log('/hello --> ' + nodes[0].value)
  })
})

[tdjsnelling]

Hm, that works no problem. Can't see anything hyper related with `DEBUG=`.

• OS: macOS Mojave 10.14.4
• Node: 11.6.0
• hyperdb: ^3.5.0

[tdjsnelling]

When I try and switch that example to filesystem storage, it fails:

/Users/tsnelling/dbtest/index.js:19
    console.log('/hello --> ' + nodes[0].value)
                                         ^

TypeError: Cannot read property 'value' of undefined
    at clone.get (/Users/tsnelling/GitHub/dbtest/index.js:19:42)
    at GetRequest._finalize (/Users/tsnelling/GitHub/dbtest/node_modules/hyperdb/lib/get.js:87:8)
    at finalize (/Users/tsnelling/GitHub/dbtest/node_modules/hyperdb/lib/get.js:222:7)
    at process.internalTickCallback (internal/process/next_tick.js:72:19)

[joehand]

Hmmm, strange! Can you share your code? Can you read the value from the original db? This seems to be working for me:

const hyperdb = require('hyperdb')
const pump = require('pump')

const db = hyperdb('./db')

db.put('/hello', 'world',  (err) => {
  if (err) return console.log(err)
  db.get('/hello', (err, nodes) => {
    if (err) throw err
    console.log('db get /hello --> ' + nodes[0].value)
  })
})

db.on('ready', () => {
  const clone = hyperdb('./db2', db.key) 

  const stream = db.replicate()
  pump(stream, clone.replicate(), stream, err => {
    if (err) console.error(err)
  })

  clone.get('/hello', (err, nodes) => {
    if (err) throw err
    console.log('clone /hello --> ' + nodes[0].value)
  })
})

See https://glitch.com/edit/#!/lightning-subway?path=example.js:2:0

[joehand]

Feel free to hop in https://gitter.im/datproject/discussions for some more interactive debugging help.

[tdjsnelling]

Through trial and error I have solved the problem - when creating a hyperdb with a hardcoded key, I could not replicate the db using that same key. But when creating a db with no key, getting the generated key with db.key and using that to connect a peer, replication works no problem!

[tdjsnelling]

That said, replication only seems to work one way... if I db.put on a peer, should that not be reflected in the original hyperdb?

[joehand]

Through trial and error I have solved the problem - when creating a hyperdb with a hardcoded key, I could not replicate the db using that same key. But when creating a db with no key, getting the generated key with db.key and using that to connect a peer, replication works no problem!

Can you share the code for this? Would be great to add as a test case.

That said, replication only seems to work one way... if I db.put on a peer, should that not be reflected in the original hyperdb?

Did you authorize that peer? If not, the changes will not be replicated on the network. If you did, may be a bug!

[tdjsnelling]

Sure thing, code is at https://github.com/tdjsnelling/dat-keyserver. Oops, I have not authorised the peer, that'll be why!

[joehand]

Awesome, thanks! Would you be interested in adding a bit of documentation on db.replicate for the network and authorization bits?

[tdjsnelling]

Sure no problem. I'm gonna go ahead and close this issue, will submit a PR when I get some time to write something up.