ANNOUNCEMENT: Plans for the next major version 13.0.0

[pubkey]

Plans for the next major version

So in the last major RxDB versions, the focus was set to improvements of the storage engine. This is done. RxDB has now multiple RxStorage implementations, a better query planner and an improved test suite to ensure everything works correct. This let to huge improvements in write and query performance and decreased the initial pageload of RxDB based applications.

In the next major version, I want to focus on the replication protocol. When I first implemented the GraphQL replication a few years ago, I had a specific use case in mind and designed the whole protocol and replication plugins around that use case.

But the time has shown, that the current replication protocol is a big downside of RxDB:

• The replication relies on the backend to solve all conflicts. This was easy to implement into RxDB because the whole responsibility was given away to the person that has to implement a compatible backend.
• In each point in time, the replication does either push or pull documents, but never in parallel. This slows done the whole replication process and makes RxDB not usable for the implementation of features like multi-user-real-time-collaboration or when many read- and write operations have to happen in a short timespan.
• After each push, a pull has to be run to check if the backend had changed the state to solve a conflict.
• The replication protocol does not support attachments.

So I want to replace the whole replication plugins with a new replication protocol. The main goals are:

• Push- and Pull in parallel.
• Use the data in the changestream (optional) to decrease replication latency.
• Implement the conflict resolution into RxDB so that the client resolves it's own conflicts and does not rely on the backend.
• Decrease the complexity for a compatible backend implementation. The new protocol relies on a dumb backend. This will open compatibility with many other use cases like implementing Offline-First in Supabase or using CouchDB but having a faster replication compared to the native CouchDB replication.
• Make it possible to use CRDTs instead of a conflict resolution.

How will the new protocol work?

On the RxDocument level, the replication works like git, where the fork contains all new writes and must be merged with the master before it can push its new state to the master.

• The client pulls the latest state from the master.
• The client does some changes
• The client pushes these changes to the master by sending the latest known master state and the new client state of the document.
• If the master state is equal to the latest master state of the client, the new client state is set as the latest master state.
• If the master also had changes and so the latest master change is different then the one that the client assumes, a conflict arises that is solved by RxDB. The resolved conflict causes a new write to the document which will trigger the push again.

This works great with the already existing RxDB revision handling. A beta of the whole protocol is already implemented and used in the Memory Synced RxStorage. You can read the code of the replication here.

Will an old RxDB compatible backend work with the new protocol?

No. It will not work out of the box. But there will be ways to easily make it compatible and it will still be possible to resolve conflicts on the server side.

Please discuss.

Other planned changes

[WonderPanda]

Really excited about this one! I think that it could be a huge game changer when it comes to the types of applications we can build and adoption overall. Thanks for all the hard work you've put into RxDB

[tjbo]

Would be really great if we could add authentication (just being able to attach custom headers).

Right now I'm using syncGraphQl replication command, and from what I've read and looked at there is zero security on this, and no easy way to implement.

Just my 2 cents, great library otherwise.

[tjbo]

NVM my other comment, I just found:

 replicationState.setHeaders({
    Authorization: `...`
});

[cetsupport]

Regarding replication, here is one scenario we like to replicate via multiple protocols in IoT domain. For example:

Real time telemetry data should replicate via websocket subscription. Meanwhile, the history data like to use REST API.

We need merge data from those two parts together in right order. Suggest add this feature to roadmap.

[chrisdrackett]

I'm excited about the focus on replication. Our biggest pain-point on RxDB is that we need to keep managing our postgres + graphql server. It existed from before we started using RxDB, but it takes a lot of effort to keep it going. I've been thinking about moving to couch if only because it seems like managing the server would become much more work, but sounds like it might be a good idea to hold off on these plans until I know more about how this all is going to work.

[chrisdrackett]

@pubkey curious what the initial targets will be for a backend here. Will the focus still be on couch and graphql or something else?

[johannesfritsch]

+100 for local conflict resolution

[pubkey]

You can now try out the latest beta of RxDB 13.0.0. https://github.com/pubkey/rxdb/blob/master/docs-src/releases/13.0.0.md