Dexie in Node

[ghost]

What's a database solution most similar to Dexie which runs in Node? In 2017, I beleive that there is still no implementation of IndexedDB which runs in Node. I love the Dexie API. Have your say.

[ghost]

David, how involved do you think it would be to fork Dexie to use LevelUp instead of the IndexDB API directly? LevelUp runs both in Node and in the browser. I still don't think that any of the IndexedDB implementations in Node use LevelUp even - they use SQL engines. That just seems weird.

[dfahlander]

I'm definitely on the path to provide the Dexie api to node. The question is which database stack to build it on. During the very last weeks, much work has also been done by @brettz9 to finalize a node port for IndexedDBShim. But I would also like to create a Dexie that supports node natively. Anyhow, the work of breaking out Dexie api from its expression engine has already started (branch next-expression-engine). The plan I have is to create an isomorphic database with conflict free sync (explained briefly here).

I've so far been leaning on building the first non-indexedDB based Dexie on some stable database stack with lots of community support, existing tooling, great performance and scalability etc. It would be great if it already had full transaction support as well and Json document features. I've been looking much into postgres so far. It has moved towards noSQL features with indexing JSON docs while at the same time being fully transactional. It is also capable of running javascript based stored procedures using a plugin for that (don't recall the name of that plugin).

LevelDB on the other hand, is more pure but as I understand, to the cost of tooling support and in effect also performance (I'm assuming). I might be wrong and uninformed though. There might be components in the level community that are hugely utilized that I don't know of yet. However, IndexedDB maps quite well to an SQL model, except for the less stricter schema. The JSON support in postgres on the other hand, is quite close to the indexedDB model.

LevelUp is very low level compared to indexedDB. Same btree features, but no transactions, no shared db connections, no types, no indexes. Much work would have to be done to adapt it to Dexie or indexedDB. A node implementation of indexedDB based on LevelUp would be interesting though. I actually think that Chromium does that in practice, but using c++ level api. Maybe that code should be ported to a generic node module... That's something that would probably be a more 'native' indexedDB for node. But still - no tooling, no db admin tools, backup, restore, query tool, reindex, sharding utility, etc. That's why I'm leaning on building a Dexie api that uses something like postgres as "database driver" instead.

[simplygreatwork]

This is a tiny library for levelup inspired by Datomic. https://github.com/eugeneware/level-immutable http://www.datomic.com/

I'm not sure if an approach like this would help you with levelup transactions for Dexie. But it is kind of cool.

[simplygreatwork]

These exist as well: https://github.com/eugeneware/level-transaction https://github.com/cshum/level-transactions

[acarl]

I definitely think you are on the right track by looking at PostgreSQL first. Dexie maps to a SQL database very well. PostgreSQL offers you everything you would need from the start and has great community support.

[aral]

What’s the current state of development on this? A LevelDB-backed Node implementation would also allow synchronisation to be added in-process for single-tenant (personal) federated/peer-to-peer web sites/apps. It’s possible that I might end up going this route for implementing data sync for Indie Site (https://indienet.info), based on a causal tree style directed acyclic graph implementation (my goal is not to replicate the whole data store on every node but ensure that any nodes that have the same set of operations commute on the same view of that data).

[aral]

Quick update: the only solution I could find so far for an isomorphic data layer (apart from PouchDB – which I’ve discounted for other reasons), is NanoSQL (https://docs.nanosql.io/adapters). They’re IndexedDB in the browser and LevelDB in Node. As much as I love what I’m seeing of Dexie’s API and workflow, it looks like I might have to go with NanoSQL despite the SQL. Can’t imagine having two separate data layers on client/server in an otherwise isomorphic app and IndexedDBShim doesn’t support an in-process database in Node either.

[dfahlander]

The state of the development is that I've been busy almost every single non-customer hour + all late nights and weekends the last year, analysing, researching, designing and poc:ing an isomorphic database, based on indexedDB at client and Postgres or Citus on server, optionally protected by a server side access control layer. This time I'll be going to commercialize it partly. By commercializing some crucial parts of the server side features, I hope to finance myself to work more on Dexie and the new Isomorphic database and less with consultancy the years to come. I'm hoping to show some results in autumn 2018. I've been cooking some innovative features that I think could gain some traction in the community, we'll see.

@aral Suppose you need something right now and my stuff isn't going to be released until autumn earliest. I've also looked at NanoSQL and find the idea kind of interesting, given the absence of a pure javascript based, typescript-friendly query language (which is one of ideas I have from my part).

If I'd have to create a true isomorphic db app today, I would also consider NanoSQL. If relational features wasnt cruzial, I'd definitely go for pouchdb.

Question: Why is in-process database important for your project? Doesn't indexeddbshim support that through sqlite?

[aral]

Hey @dfahlander, thank you so much for the detailed response + I hear you completely. Your plan sounds like a good one (including the roadmap for sustainability). Also, thank you for looking at NanoSQL and for sharing your thoughts on it; it’s really valuable validation.

My reasons for not considering Pouch are two-fold: firstly, the long-polling (although some web socket implementations exist, they don’t seem to be actively used/supported) and secondly, that it does a lot of what I will be doing anyway regarding replication but in a different way: I’m exploring sparse replication that works similar to the way web sites work today – by providing only the data you need, instead of trying to sync the full store – but in a peer-to-peer topology. I’m also looking into using a directed acyclic graph (DAG)/Merkle DAG-style data type with public-key cryptography for authorisation and authentication and nodes with end-to-end encrypted content for access control.

You’re right that indexedbshim does support sqlite; I’d overlooked the fact that sqlite is in-process (after having used it countless time on iOS – yay me!) I will run some tests today to test the performance of that vs. NodeSQL with LevelDB :)

Regarding why an in-process database is important for my project: I’m building a single-tenant (there is no concept of “users” only one owner per site) federated personal web site platform and one of the goals is that it is as lightweight and easy to install as possible. The long-term goal is to get people up and running with their own indie sites at their own domains in as much time as it currently takes to sign up for a Facebook account. The difference being that it is a place you own and control in a free and open, decentralised/federated, and interoperable network instead of a place that you rent from some corporation that makes money by profiling you :)

I absolutely love Dexie’s API so even if I end up going with NanoSQL, I might actually implement a façade on top of NanoSQL as a compromise for now.

Thank you again for taking the time to share your thoughts, plans, and feedback and, of course, for making and sharing Dexie. It is a beautiful work of art and I can’t wait to see where you take it next. All the best with making it a sustainable project and please let me know if there’s ever anything I can do to help you promote it, etc.

[dfahlander]

Thanks a lot for all the nice feedback :-)

I'll be using github tickets on Dexie.js to post news regarding the new Dexie and it's isomorphic API when it's time.

I wish you all best and a success for Indienet!

[samratarmas]

Hello @dfahlander I have been following through all your mentions about isomorphic dexie in several dexiejs open and closed issues, and would love to hear about whether it is still something you are working on.

[dfahlander]

I'm definitely working on it a lot since years back - and will release a new library soon (what is soon? :-> In my world, it is hopefully winter/spring 2020) that will be kind of a new Dexie major version but with a different name since it's not directly tied to IndexedDB, like Dexie is. Except for its indexeddb support, there will be providers for PostgreSQL and memory, and a generic provider interface. Dexie will continue being maintained and the libraries will coexist. I'm full of inspiration and eagerness of sharing all the ideas, but have waited, and will continue waiting til the API is fully set, but that is very close now.

[bebraw]

fake-indexeddb allows you to replace Dexie's IndexedDB bits with an in-memory based solution. That's enough for testing at least and may be a possible direction for anyone who might want to persist the data.

@dfahlander Good luck with the release of the new library. Thanks for your work on Dexie. 👍

[kotasudhakar]

fake-indexeddb allows you to replace Dexie's IndexedDB bits with an in-memory based solution. That's enough for testing at least and may be a possible direction for anyone who might want to persist the data.

@dfahlander Good luck with the release of the new library. Thanks for your work on Dexie. 👍

I understand, recently i have implemented dexie js db to store data alongside using Fake-indexedDB in the electron main process (not render process) where i can't use storage manager API as it uses navigator which is a browser only thing. So since i am now using electron main process(node) how can i make the storage persisted. Have tried to search for last couple of days, not able to get anything regarding making storage persisted in node(main process) Please help @dfahlander @aral @bebraw @samratarmas

[dfahlander]

IndexedDBShim by @brettz9 should emulate indexedDB on node and store it persistently into an SQLite database. https://github.com/indexeddbshim/IndexedDBShim?tab=readme-ov-file#node-set-up

[frankleng]

https://nodejs.org/docs/latest/api/sqlite.html sqlite might just be shipping w/ future versions of node after all