ANNOUNCEMENT: Planned changes for the next major version of RxDB

[pubkey]

Based on the last user survey, I have worked out some changes that I want to make in the next major release of RxDB. Please start discussions and improvements in this issue.

Move rxjs into a plugin instead of having it internal

RxDB relies heavily on rxjs. This made it easy in the past to handle the data flow inside of RxDB and also created feature-rich interfaces for users when they want to observe data. As long as you have rxjs in your project anyways, like you would have in an angular project, there is no problem with that. As soon as a user has another data-handling library like redux or mobx, rxjs increases the build size by 22kb (5kb gzipped) and also adds the burden to map rxjs observables into the own state management.

The change would ensure that rxjs is no longer used inside of RxDB. And also there will be a RxDB-plugin which offers the same observable-features as there are today, but optional. This would also allow us to create plugins for mobx or react-hooks in the future.

Move pouchdb into a plugin

When I started creating RxDB, I used the best solution for a noSQL storage engine that I could find. This was pouchdb. Not only because it is a very mature project, but also because it has adapters for so many environments. The problem with pouchdb is the build size of 30kb (gziped, with indexeddb adapter) and also the performance decrease by it's overhead which comes from how it handles it's revision tree.

The change would ensure that the storage-engine is abstracted in a way that I can be swapped out by any other noSQL database out there. Users could then use different storage engines, depending if they want smaller builds or better performance. For example there are these out there:

• NeDB
• Minimongo
• Dexie.js
• Worker-Pouch.
• Or any database with a changestream like Postgre, mongo etc.

This would also make it possible to use RxDB together with NativeScript in the future. Pouchdb will still be the default in the main build.

Make RxDcouments immutable

At the current version of RxDB, RxDocuments mutate themself when they recieve ChangeEvents from the database. For example when you have a document where name = 'foo' and some update changes the state to name = 'bar' in the database, then the previous javascript-object will change it's own property to the have doc.name === 'bar'. This feature is great when you use a RxDocument with some change-detection like in angular or vue templates. You can use document properties directly in the template and all updates will be reflected in the view, without having to use observables or subscriptions.

However this behavior is also confusing many times. When the state in the database is changed, it is not clear at which exact point of time the objects attribute changes. Also the self-mutating behavior created some problem with vue- and react-devtools because of how they clone objects.

Also, to not confuse with fast changes that happen directly after each other, the whole json-data-to-RxDocument-piple has to be synchronous. With the change, this can be async which will allow us to have async post-result-transformations, like an asynchronous encryption plugin (with the Web Crypto API) or also move things into a webworker.

The change would make all RxDocuments immutable. When you subscribe to a query and the same document is returned in the results, this will always be a new javascript object.

Use immutable objects instead of deep-cloning stuff

RxDB often uses outgoing data also in the internals. For example the result of a query is not only send to the user, but also used inside of RxDB's query-change-detection. To ensure that mutation of the outgoing data is not changing internal stuff, which would cause strange bugs, outgoing data is always deep-cloned before. This is a common practice on many javascript libraries. The problem is that deep-cloning big objects can be very CPU expensive. So instead of doing a deep-clone, RxDB will assume that outgoing data is immutable. If the users wants to modify that data, it has be be deep-cloned by the user. To ensure immutability, RxDB will use deep-freeze in the dev-mode (about same expensive as deep clone). Also typescript will throw a build-time error because we will use ReadonlyArray and readonly to define outgoing data immutable. In production-mode, there will be nothing that ensures immutability.

[pubkey]

I'm closing this and I moved the content to the backlog. The next major release will be published soon.

[jshearer]

Hi @pubkey, sorry for bumping this year-old issue. I'm interested in two parts: moving rxjs out of core, and abstracting away the core dependence on pouchdb.

My company has been using RxDB for a while now. We're essentially building a platform for users (us, at first) to quickly build apps that work offline and can react in realtime to data changes (sort of like 2-way data binding, but against a clientside syncable snapshot of their data). In order to do this, we have written a fairly extensive RxDB/MobX compatibility layer to allow observing any data changes automatically, including across relationships and inside nested arrays.

Now, this compatibility later is currently fairly tightly integrated with the rest of our platform, and it probably reaches into too many undocumented places in RxDB/pouchdb for it to be just polished/published as is, but in your "roadmap" above you mentioned that once rxjs is extracted, that opens up the possibility for other similar observability integrations, such as with mobx.

So, have you given any more thought to how the API for object change reactions will look? We're excited about writing the MobX integration!

We are also excited about moving away from pouchdb as a dependency. In order to get RxDB working with any semblance of performance in React Native, we are using pouchdb-adapter-react-native-sqlite, which does work, until you run into a bug with pouchdb-adapter-websql-core which was kindly extracted into its own module when it became clear that the pouchdb team no longer had the bandwidth to maintain it. Ultimately we would love to run on something like IndexedDB, though at the moment there doesn't appear to be any native modules for indexeddb, so that might be another area we could contribute to this whole ecosystem.

[pubkey]

@jshearer no problem opening this up.

So the plan I had in mind, was to remove all usage of internal rxjs out of RxDB. Instead of using rxjs internally, we could switch to EventEmitters which are much more slim and basic. Instead of having an observable, you could call addEventListener() like you do with click events in the browser. On top of that, people can develop any plugins that add functionality for rxjs, mobx or react hooks. But this requires a big rewrite of RxDB internals which is why I havent done it.

To fully move pouch out of rxdb, we have to move any call to the pouchdb API into the RxStorage. I started doing that here but there are still many call directly into pouchdb for multiple reasons. Moving stuff into the RxStorage must be done really carefully because once RxStorage is final, other databases will be wrapped into it and it will be hard to change stuff. Once we have done this, we can create other implementations of the RxStorage interface for indexeddb or whatever database that supports the basic features.

If you want to start working on that, I think we should start with removing rxjs since it is way easier and straightforward.

[jshearer]

So the plan I had in mind, was to remove all usage of internal rxjs out of RxDB. Instead of using rxjs internally, we could switch to EventEmitters which are much more slim and basic. Instead of having an observable, you could call addEventListener() like you do with click events in the browser. On top of that, people can develop any plugins that add functionality for rxjs, mobx or react hooks. But this requires a big rewrite of RxDB internals which is why I havent done it.

Yes, this sounds great, and it actually sounds like it'll clean up/simplify a lot of things that might be unnecessarily using rxjs just because it was there (like migrations 😁 ).

Understanding that this is an open source project and that you probably have a day job and family and all that, can you say anything about the timeline or prioritization of that effort? It sounds like you haven't started yet, so probably a while? No pressure, just trying to plan myself.

[pubkey]

@jshearer I'm sorry to say that, but I use angular in most of my projects. There I am bound to rxjs anyway so I have no big incentives to do this big efford.

For my side projects I am even using RxDB on the server side. There I will have to replace pouchdb with something faster one day, then I will change that on RxDB. But it is unlikely that this will happen in 2020.

But if you are able to work on that in company time, maybe you should start :)

[jshearer]

Okay, thanks for the info. It's good to know that if I were to put effort into this, it wouldn't be duplicating your effort.

But if you are able to work on that in company time, maybe you should start :)

I might just try and do that...

[adampash]

Is removing rxjs no longer a goal of the project? Just curious, since RxDB seems to have the most robust storage adapters by a longshot for JS, but the rxjs requirement is a heavy addition if it's not needed. E.g., I use mobx, which fits my needs extremely well, so it would be hard to transition everything to rxjs. Additionally, I assume there's a fair amount of extra overhead mapping all the observables between rxjs and mobx.

Anyway, this is a very impressive library and this is not at all a complaint. Just curious to hear if there are any changes or updates on this issue (which I realize is also closed, so I understand if I get no responses).

[pubkey]

No rxjs is not heavy compared to anything else we could use. So the goal of removing it was dropped from my side because I think it is not worth the effort. A PR to have rxjs observables as a plugin would still be accepted.

With the custom reactivity feature you can make RxDB returning mobx observables by your own: https://rxdb.info/reactivity.html