orbitdb / field-manual

The Offical User's Guide to OrbitDB
208 stars 43 forks source link

How does orbitdb sync? #68

Open subhasisbanik opened 5 years ago

subhasisbanik commented 5 years ago

I have prepared a cluster of 2 nodes in IPFS with IPFS Daemon and swarm key. I am running orbitdb and connecting to the IPFS node with "ipfs-api" in HTTP protocol. I am able to post data through orbitdb (feed, keyvalue) from one node and am able to fetch it back in another orbitdb instance connected to the other node. The issue is that when I am calling a "put" in one instance and trying to "get" it from another instance, its not able to "get" it the first time. But if the same "get" is being called from the same node the second time, its showing the data.

I am not sure how orbitdb sync works but below is the code that I am running.

Node 1----------- const OrbitDB = require('orbit-db'); const IpfsApi = require('ipfs-api'); const ipfs = IpfsApi('localhost', '5002', {protocol: 'http'}) const orbitdb = new OrbitDB(ipfs); const writePermissions = [orbitdb.key.getPublic('hex'), ]; const kv = await orbitdb.kvstore('new-db',{write:writePermissions}); await kv.load(); await kv.put('test', 80);

Node 2------------ const OrbitDB = require('orbit-db'); const IpfsApi = require('ipfs-api'); const ipfs = IpfsApi('localhost', '5002', {protocol: 'http'}) const orbitdb = new OrbitDB(ipfs); const db = await orbitdb.open('/orbitdb/QmPJ4QXJDfBjPKB6JKZcqmrDfDxbubC85BMQvxDcR2Gunb/new-db'); await db.load(); db.get('test');

Please help!!!

tyleryasaka commented 5 years ago

@subhasisbanik I know what the cause of this issue is (I ran into the same thing myself). db.load() resolves only after the database has finished loading from storage into memory. It does not wait for the database to be replicated from peers. The replication happens in the background, and is likely completing a split second after db.get('test') is run. That's why when you run the code a second time, you get the expected value.

If you want to wait for the replication to finish, you can listen for the replicated event. E.g.

await Promise.resolve(resolve => {
  db.events.on('replicated', resolve)
})

Beware, though; if no replication occurs, the above code will hang indefinitely. (For example, if the peers are already fully in sync.) This is a major problem; I don't know of a good solution off the top of my head, though I am looking into options. I'll report back if I find anything.

subhasisbanik commented 5 years ago

Thanks for the explanation. I would surely wait for a reply from you..

On Tue, Nov 20, 2018, 10:06 AM Tyler Yasaka <notifications@github.com wrote:

@subhasisbanik https://github.com/subhasisbanik I know what the cause of this issue is (I ran into the same thing myself). db.load() resolves only after the database has finished loading from storage into memory. It does not wait for the database to be replicated from peers. The replication happens in the background, and is likely completing a split second after db.get('test') is run. That's why when you run the code a second time, you get the expected value.

If you want to wait for the replication to finish, you can listen for the replicated event. E.g.

await Promise.resolve(resolve => { db.events.on('replicated', resolve) })

Beware, though; if no replication occurs, the above code will hang indefinitely. (For example, if the peers are already fully in sync.) This is a major problem; I don't know of good solution off the top of my head, though I am looking into options. I'll report back if I find anything.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/orbitdb/orbit-db/issues/474#issuecomment-440135153, or mute the thread https://github.com/notifications/unsubscribe-auth/AYPkaAqZ7xiYgFJYQ7_nQtulBRrtfNWuks5uw4bHgaJpZM4XR1S9 .

tyleryasaka commented 5 years ago

Hi @subhasisbanik I did find a solution, which requires a couple modifications to orbit-db.

First, I made this change to orbit-db: https://github.com/projectaspen/orbit-db/commit/f91220d3b5a8de38389464675a4db5e00319f180

Then I made this minor adjustment to orbit-db-store: https://github.com/projectaspen/orbit-db-store/commit/a31e1f728c8edd0395b2722766a0d181e3f088eb

This allows me to wait for the new synced event which is fired once for each peer + database, after all of the data has been copied over. If you wait for this event (once after opening each database) for each node you want to sync with, then when you retrieve data for orbit-db, it will be up-to-date.

To make this process smoother, I created 2 packages: 1 for the orbit-db server node, one for the client. They're experimental right now, but feel free to check them out! They use the orbit-db repo changes I linked to above.

subhasisbanik commented 5 years ago

Awesome. I will revert once I have a look into it.

On Wed, Nov 21, 2018, 3:07 PM Tyler Yasaka <notifications@github.com wrote:

Hi @subhasisbanik https://github.com/subhasisbanik I did find a solution, which requires a couple modifications to orbit-db.

First, I made this change to orbit-db: projectaspen@f91220d https://github.com/projectaspen/orbit-db/commit/f91220d3b5a8de38389464675a4db5e00319f180

Then I made this minor adjustment to orbit-db-store: projectaspen/orbit-db-store@a31e1f7 https://github.com/projectaspen/orbit-db-store/commit/a31e1f728c8edd0395b2722766a0d181e3f088eb

This allows me to wait for the new synced event which is fired once for each peer + database, after all of the data has been copied over. If you wait for this event (once after opening each database) for each node you want to sync with, then when you retrieve data for orbit-db, it will be up-to-date.

To make this process smoother, I created 2 packages: 1 for the orbit-db server node, one for the client. They're experimental right now, but feel free to check them out! They use the orbit-db repo changes I linked to above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/orbitdb/orbit-db/issues/474#issuecomment-440597921, or mute the thread https://github.com/notifications/unsubscribe-auth/AYPkaClMoTPUZRD71BejXmN3D8QioSaAks5uxR7cgaJpZM4XR1S9 .

aphelionz commented 4 years ago

Moving this to the Field Manual repo so we can detail this in the book