search

ueberdosis / hocuspocus

The CRDT Yjs WebSocket backend for conflict-free real-time collaboration in your app.
https://tiptap.dev/docs/hocuspocus/introduction
MIT License
1.12k stars 108 forks source link

Ordering of YJS updates per document #536

Open martinboksa opened 1 year ago

martinboksa commented 1 year ago

The problem I am facing Updates from YJS can be handled in the wrong order due to Promises in NodeJS. The hook chain beforeHandleMessage and onChange don't wait if the previous update was already applied to the document.

I understand that YJS may apply updates in the wrong order, and although everything is okay when all updates are applied but in our case, we need to implement user email notifications that rely on ordered updates. Otherwise, we would have to think about a database structure that can handle 'unordered updates', which is more challenging and time-consuming as it requires covering all possible cases. This is not just about notifications, as we also have other structures that rely on changes.

The solution I would like Let's update the hooks beforeHandleMessage and onChange to ensure the order is guaranteed (per document). For example, when Hocuspocus receives update messages 1 (UM1) and 2 (UM2), UM2 should wait until beforeHandleMessage(UM1) and onChange(UM1) are completed before handling the next update message.

janthurau commented 1 year ago

hmm, at first it makes sense, but what is the wrong / right order? Depending on the connection speed of the client or if client is offline (and online again), the message order on the server will not be 'right' probably as well. Waiting on the server for messages to be applied in the order that they were received (by the server) would delay the user experience as changes wouldn't be applied simultaneously anymore but each change would just apply one after the other.

Not sure what we can/should do here 🤔 But about your task, not sure what kind of data you send, but probably it makes sense to anyway debounce the action that you are doing by X and then somehow group the changes together? Maybe you can somehow work around this?

martinboksa commented 1 year ago

@janthurau, at first, thank you for the quick response.

hmm, at first it makes sense, but what is the wrong / right order? Depending on the connection speed of the client or if client is offline (and online again), the message order on the server will not be 'right' probably as well.

The messages from one client will always arrive in the correct order, as guaranteed by the WebSocket design. It is impossible for one update to overtake another.

Waiting on the server for messages to be applied in the order that they were received (by the server) would delay the user experience as changes wouldn't be applied simultaneously anymore but each change would just apply one after the other.

In my opinion, it should work fine. However, if it creates a "bad UX" for users, we should look for ways to improve it.

Not sure what we can/should do here 🤔 But about your task, not sure what kind of data you send, but probably it makes sense to anyway debounce the action that you are doing by X and then somehow group the changes together? Maybe you can somehow work around this?

Fortunately, there is a workaround. We cannot use "beforeHandleMessage," and "onChange" should send updates to the message queue (each document has its message queue) (even in our case we must listen on 'yjs transaction + change' to read deleted data)

In general, I believe that this update would help simplify the hocus-pocus architecture and make it easier to solve problems related to updates. You can try to find a way to implement notifications without guaranteed order, it would be nearly impossible. Similarly, when we implemented page backlinks, it would be difficult to do without considering order. Additionally, we have two other things that depend on order.