About offline synchronization

[nieyuyao]

I needs to add a feature about offline synchronization. The scene is as follows, A and B are collaborating. A goes offline and performs some operations, and then closes the browser. I will store these ops in indexedDB. These operations will be read first after A open browser at next time. How should I deal with these ops, send them directly to the server or resolve the conflict locally first?

[alecgibson]

This is something I've meant to try, but still haven't gotten around to. I think this is in theory possible, but there's something that would need "fixing" first: when you go offline, you have your doc.data already transformed by all of the local ops you've submitted, but which haven't yet been acknowledged (or even received) by the server.

You can theoretically resubmit the ops you have in pendingOps, but data has already been transformed by them, so you'd need to find some way of resetting your data back to before it had any pending ops (we could potentially track this when you call submitOp, for example, which might also make rollbacks a bit easier).

In theory, if we could revert data, this is what the steps would look like (I think):

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)
4. Store these fields from doc:
   • collection
   • id
   • version
   • type
   • data — would need to revert this to before any ops were submitted
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. Create a snapshot that looks like: snapshot = {v: storedVersion, data: storedData, type: storedType}
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

As I said, though, I don't think this is possible without some change to sharedb that would enable us to get data before the ops were submitted. I'm more than happy to be proven wrong (@ericyhwang I think we've vaguely discussed this in the past?).

That being said, I don't think it would take a lot of work to make sharedb friendly to having sessions closed and restored (would even be nice to add some methods that basically do the above steps for us), and Pull Requests are always more than welcome!

[nieyuyao]

This is something I've meant to try, but still haven't gotten around to. I think this is in theory possible, but there's something that would need "fixing" first: when you go offline, you have your doc.data already transformed by all of the local ops you've submitted, but which haven't yet been acknowledged (or even received) by the server.

You can theoretically resubmit the ops you have in pendingOps, but data has already been transformed by them, so you'd need to find some way of resetting your data back to before it had any pending ops (we could potentially track this when you call submitOp, for example, which might also make rollbacks a bit easier).

In theory, if we could revert data, this is what the steps would look like (I think):

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)

4. Store these fields from doc:

   • collection
   • id
   • version
   • type
   • data — would need to revert this to before any ops were submitted
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. Create a snapshot that looks like: snapshot = {v: storedVersion, data: storedData, type: storedType}
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

As I said, though, I don't think this is possible without some change to sharedb that would enable us to get data before the ops were submitted. I'm more than happy to be proven wrong (@ericyhwang I think we've vaguely discussed this in the past?).

That being said, I don't think it would take a lot of work to make sharedb friendly to having sessions closed and restored (would even be nice to add some methods that basically do the above steps for us), and Pull Requests are always more than welcome!

Thanks for your reply! Let me see.🤔

[nieyuyao]

Is there another solution? The steps would look like:

1. Client goes offline.
2. Client makes some edits.
3. We only store the ops generated by editing and current doc version.
4. Client reopens browser.
5. We find a way to pull the missed ops from the remote, such as fetchOps(from, to)?
6. Stored ops transform the pulled ops.
7. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

What do you think? @alecgibson

[alecgibson]

The important thing is to get the doc back to how it was before you submitted any ops, which I guess you could do with connection.fetchSnapshot(). So amending my above:

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)
4. Store these fields from doc:
   • collection
   • id
   • version
   • type
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. NEW STEP: Fetch snapshot: connection.fetchSnapshot(collection, id, storedVersion, (snapshot) => {...})
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

I think in theory that should work, but as I say it's completely untested. Would love to hear how you get on!

[nieyuyao]

Okay, I understand. So is it necessary to perform a lock operation when the user opens different tabs? If the user has opened tab A and tab B. Do we need to find a way to allow only one page in A and B to operate when the network is disconnected?

[nieyuyao]

Is there another solution? The steps would look like:

1. Client goes offline.
2. Client makes some edits.
3. We only store the ops generated by editing and current doc version.
4. Client reopens browser.
5. We find a way to pull the missed ops from the remote, such as fetchOps(from, to)?
6. Stored ops transform the pulled ops.
7. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

What do you think? @alecgibson

Is this solution unworkable or complicated?

[alecgibson]

Is this solution unworkable or complicated?

It's just unnecessary — ShareDB will handle the missed ops for you as soon as you try to submit the pending ops you have stored.

[alecgibson]

So is it necessary to perform a lock operation when the user opens different tabs?

You shouldn't need to lock documents. Again, ShareDB should handle the merge conflicts for you, just as if you were collaborating in real time.

[alecgibson]

(Although obviously the longer you've been offline, the less the automatic merges may make sense)

[nieyuyao]

So is it necessary to perform a lock operation when the user opens different tabs?

You shouldn't need to lock documents. Again, ShareDB should handle the merge conflicts for you, just as if you were collaborating in real time.

Tabs A and B have performed some ops each other after offline. Both tabs were closed at a later time. Then we connect internet and open tab C. At this time, the ops stored by A and B will be mixed and submit(doc.submitOp). There may be conflicts between mixed ops. Can we call doc.submitOp directly?

[alecgibson]

Aha I see. The "cleanest" way to deal with it is probably:

• store each state (A & B) separately
• for each stored state attached to a document, create a new Connection, where you can have independent docs
• run the steps I outlined above for each connection
• ShareDB should still handle the merge conflicts

[nieyuyao]

The important thing is to get the doc back to how it was before you submitted any ops, which I guess you could do with connection.fetchSnapshot(). So amending my above:

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)

4. Store these fields from doc:

   • collection
   • id
   • version
   • type
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. NEW STEP: Fetch snapshot: connection.fetchSnapshot(collection, id, storedVersion, (snapshot) => {...})
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

I think in theory that should work, but as I say it's completely untested. Would love to hear how you get on!

Thank you very much! I try this plan. 🙌🏻

[nieyuyao]

It is troublesome to fetch different documents when syncing offline ops. There may be another solution, but I don’t know if it can work.

1. Store the clientID (src) and seq field at the same time when storing op. The clientID field of A and B are different.
2. When a new tab is opened, we may have the following op:

clientA: op1, op2, op3, .... clientB: op1, op2, op3, .... ....

Compose ops of same clientID.

clientA: opA clientB: opB ....

3. Create a new Connection specifically to submit these ops.

   const conn = new Connection(ws);
   const { clientID, op, v, seq } = storedOp;
   conn.send({ a: 'op', src: clientID, op,  v, seq })

[alecgibson]

I'm not sure I follow. You mean if you have two tabs open, you want to be able to synchronise between the two tabs whilst still being offline?

[wangjianweiwei]

@nieyuyao hello Are you also working as an editor. You can give me your WeChat number and learn together.

[nieyuyao]

I'm not sure I follow. You mean if you have two tabs open, you want to be able to synchronise between the two tabs whilst still being offline?

Aha, I may not express it clearly. The steps are as follows:

1. Client A goes offline.
2. Client A makes some offline edits
3. Client A close browser.
4. Store these fields from doc:
   • collection
   • id
   • version
   • type
   • pendingOps
5. NEW STEP Store these fields from connection:
   • id, which is clientID of Agent
   • seq
6. Client B reopens browser.
7. Get doc: collection.get(collection, id)
8. Subscribe newest snapshot.
9. NEW STEP Create a new Connection specifically to submit these offline ops A edited.

   const ws = new ReconnectingWebsocket(url);
   const conn = new Connection(ws);
   const { pendingOps, version, id, collection, seq, clientID } = stored;
   conn.send({
   a: 'op', 
   src: clientID, 
   op: pendingOps, 
   v: v,
   seq,
   c: collection,
   d: id
   })

   Is the process of synchronizing these ops just like A is still online?

[alecgibson]

I don't understand the motivation for this? You mean Client B is just Client A, but reconnected? As I said above, I think you need to make sure you restore to the version you went offline at (not the latest version).

[nieyuyao]

The important thing is to get the doc back to how it was before you submitted any ops, which I guess you could do with connection.fetchSnapshot(). So amending my above:

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)

4. Store these fields from doc:

   • collection
   • id
   • version
   • type
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. NEW STEP: Fetch snapshot: connection.fetchSnapshot(collection, id, storedVersion, (snapshot) => {...})
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

I think in theory that should work, but as I say it's completely untested. Would love to hear how you get on!

Does inflightOp need to be stored?

[alecgibson]

You shouldn't need to if your connection has gone offline, because it gets added back onto pendingOps

[happy15]

You shouldn't need to if your connection has gone offline, because it gets added back onto pendingOps

will beforeunload trigger _onConnectionStateChanged? @nieyuyao lets test this.

[nieyuyao]

You shouldn't need to if your connection has gone offline, because it gets added back onto pendingOps

I tested it. _onConnectionStateChanged is not be triggered when closing tab. So is it reasonable to store inflightOp? In fact, we are not sure whether inflightOp has been received by the server.

[alecgibson]

If you keep a stable src and seq on your client, resending the inflightOp is fine, and is expected to happen in "normal" operation (eg on an unstable internet connection). If the op has already been received by the server, then the server will just reject it with an ERR_OP_ALREADY_SUBMITTED error, which is non-fatal.

[nieyuyao]

Aha, there are some differences between Connection.fetchSnapshot and Doc.subscribe. It's not just the difference of getting data. Agent will bind stream when calling Doc.subscribe. But Connection.fetchSnapshot is not. This will cause operations to be unable to synchronize to the other clients if use Connection.fetchSnapshot.

[alecgibson]

Yes, after ingesting the snapshot and replaying all of your ops, you'll have to resubscribe, but you can just do that with doc.subscribe().

[benjismith]

Hello! I'm just on the cusp of implementing an offline-mode for my ShareDB app, so I'd like to add a few ideas to the mix...

In my scenario, a user can have multiple docs open at the same time (for example, two different chapters in the same book, as well as the margin comments for each of those chapters), so I would need the offline operation buffer to keep track of all the pending operations for all open docs.

But taking things one step further, my users might also have other content that they need to be available offline (for example, some other chapter in the same book, which they haven't opened during their online session, but which they want to be available if they happen to go offline).

So I'd like to have an optional "pre-caching" plugin that fetches snapshots the user hasn't actually opened yet, but is likely to need during an offline session. I'm imagining something like this...

let shareDb = new ShareDB({
  db: ShareDBMongo({
    mongo : (callback) => { myMongoClient.connect(callback); }
  }),
  milestoneDb: new MongoMilestoneDB({
    mongo : (callback) => { myMongoClient.connect(callback); }
  }),
  pubsub: RedisPubsub({ client : myRedisClient }),
  precache: MyPreCacheHandler({ config : whatever })  // <--- this thing right here
});

The MyPreCacheHandler code on my end is completely capable of keeping track of which docs to pre-cache. Each user has a Workspace manifest, which keeps track of all the collection/id pairs the user is interested in including in their offline cache, as well as the timestamp of those items. It looks like this:

{
  "book/ABC123" : 1623725842,
  "chapter/XYZ789" : 1623634967,
  "chapter/LMNOP456" : 1623437522,
  "comment/XYZ789" : 1623634967
}

When the user is online, the pre-cache handler subscribes to updates on this manifest, and whenever there's an update to some key, it fetches the latest version of the relevant doc. So when the user goes offline, we know we've already pre-cached a recent snapshot of the doc we need. The only thing missing is some way for my pre-cache hander to integrate with ShareDB, so that the snapshot could be integrated into the normal doc lifecycle.

With this pre-cache handler in place, as long as the user never interacts with content outside their manifest, they can create/open/fetch any doc, and the application works exactly the user never went offline.

I don't expect every application to have similar needs regarding offline pre-caching, so it makes sense to architect it as a pluggable interface, with minimal hooks back into ShareDB core code.

For what it's worth, my application is a desktop app using a Chromium container (same basic idea as Electron), so the user can start the app, edit multiple different documents, create new content, etc... all without an internet connection. They might even have multiple consecutive sessions of the app completely in offline mode, before someday reconnecting and synchronizing their offline operation buffers.

Thoughts?

[alecgibson]

@benjismith I might be missing something, but if your client already knows which docs it cares about, can't it just subscribe to them, and then they'll be available offline through the same mechanism as everything else?

I'm not sure I understand what the precache is meant to do.

[benjismith]

Well, I'm still pretty new to ShareDB, so it's totally possible I misunderstood something...

But in my application, the user might have hundreds of individual objects in their offline cache. For example, several different books they're working on, each with a hundred or more chapters, some margin-comments, etc... If the user goes offline, we want to have all that content available for editing.

But we don't want to keep an open subscription for all those different docs (unless I've misunderstood and this is a super-lightweight kind of thing). So we use this manifest method to keep local objects up-to-date without the burden of a zillion open doc subscriptions.

Does that make sense? Or am I making bad assumptions about the cost of subscriptions?

[alecgibson]

Or am I making bad assumptions about the cost of subscriptions?

With performance, it's always best to do some tests in your particular setup and make decisions based on actual measurements.

the user might have hundreds of individual objects in their offline cache

I'm not sure I understand how a server-side plugin will help this — if they've gone offline, then surely they won't have access to the server-side cache? And from your description I'm not sure I understand how the cache works any differently to a normal subscription?

[benjismith]

Oops, you're right... I meant it was a client-side plugin. Sorry, now I see why that code-example didn't make any sense. Sorry about that 🙄

[benjismith]

Hey, I just wanted to see if anybody is working on this in the near-term... I'm willing to pay a nice bounty for a PR that implements the offline mechanisms described here :)

[nieyuyao]

The important thing is to get the doc back to how it was before you submitted any ops, which I guess you could do with connection.fetchSnapshot(). So amending my above:

1. Client goes offline
2. Client makes some offline edits
3. Client is about to close browser (eg beforeunload, although this possibly has some reliability issues)

4. Store these fields from doc:

   • collection
   • id
   • version
   • type
   • pendingOps
   • any other optional flags you may have set manually (eg preventCompose, submitSource)
5. Client reopens browser (starting with fresh session)
6. Get doc: collection.get(collection, id)
7. NEW STEP: Fetch snapshot: connection.fetchSnapshot(collection, id, storedVersion, (snapshot) => {...})
8. Ingest the snapshot: doc.ingestSnapshot(snapshot)
9. Replay all of the ops: storedOps.forEach((op) => doc.submitOp(op))

I think in theory that should work, but as I say it's completely untested. Would love to hear how you get on!

Is it possible that the same op is submitted to the server? And the server has no ability to reject the submission, because v of op is not stored.

[alecgibson]

Is it possible that the same op is submitted to the server? And the server has no ability to reject the submission, because v of op is not stored.

tl;dr: make sure you also save your connection's src and seq values.

There's a subtle difference between op "versions" and doc "versions" in ShareDB.

In general, the version of the document talks about the canonical version: that is, an op with a version has been committed to the database. For example, you can see here that the doc.version is only incremented once the op has been acknowledged by the server in doc._handleOp().

So that leaves the question of how does the server distinguish duplicate ops coming from the same client? That's handled by a pair of values you may have seen if you've investigated the op data structure: src and seq, where src is some client ID, and seq is just a monotonically incremented sequence number. Together, src and seq uniquely identify an op submitted from a client.

The server can then check for duplicate submissions, and fails with a recoverable error.

[zzph]

Could someone please clarify this thread? I'm unclear what the proper way submitting offline changes (say stored in localStorage/indexedb).

1. There is no documentation how.send works. It doesn't have a callback, and doesn't tell my 'local' object of the changes. For that, I need .submitOp
2. Should I be using .send or .submitOp. If I use .submitOp why do I need to track all those extra parameters (src,seq etc)?

My attempt:

Store the OP

{
          a: 'op',
          src: connection.id,
          op: pendingOps[0].op,
          v: version,
          seq: connection.seq,
          c: collection,
          d: id,
}

Send (?) the OP

connection.send(storedOp)

Tell local app about the change

doc.submitOp(storedOp.op)

[alecgibson]

@zzph I don't think you should need to touch .send(), which is really an internal method.

I would start by trying the steps I outlined above and let us know how you get on.

why do I need to track all those extra parameters (src,seq etc)?

See the comment directly above yours.

[zzph]

@alecgibson thanks so much. Yes I have tried that last method but am having no luck from the step 7.

The snapshot always returns null. What should the third argument be (you've named storedVersion)? I've tried passing it the entire storedOp and just the v (version) on the stored op. Still returning null

Here's my attempt:

const storedOp = {
          a: 'op',
          src: connection.id,
          op: pendingOps[0].op,
          v: version,
          seq: connection.seq,
          c: collection,
          d: id,
}

const doc = connection.get(storedOp.c, storedOp.d)

connection.fetchSnapshot(storedOp.c, storedOp.d, storedOp, (snapshot) => {
    console.log(snapshot) // <-- THIS IS NULL
    doc.ingestSnapshot(snapshot)
    doc.submitOp(storedOp.op)
})

[alecgibson]

You definitely don't want to pass the storedOp. Version should work, though: https://share.github.io/sharedb/api/connection#fetchsnapshot