any plans to support persistence for Browser like leveldb

[Volland]

Cozo wasm module is amazing but lacks persistence for in-browser apps . Support of leveldb or any browser-based persistence could open a lot of usecases

[chuanqisun]

In the interim, I've been using json-based export/import relations against browser's origin private filesystem. It isn't the fastest solution but provides the functionality. Ideally I'd love to have the ability to load and save data in binary.

[simonwh]

@chuanqisun We've done something similar, with in-memory cozo + localstorage (instead of origin private filesystem). We find it's working-ish. It's not super stable, and definitely not a long-term solution :) Would be interest to hear how you implemented it. Do you export/write file after every cozo operation? Or periodically? And then import on page load?

[chuanqisun]

@simonwh, my technique isn't ideal either. Exporting on every transaction causes performance issue. My workaround:

• Run the cozo wasm from a worker thread, so all the database I/O is isolated from the UI thread
• Export only upon user request, leading to data loss when user closes the app without saving.

In my opinion, a good continuous export implementation is a balancing act between the inevitable tradeoff between performance and data integrity. A few ideas I find useful:

• Use requestIdleCallback to let the browser pick a best moment to export but that won't work in Safari.
• Use some throttle and debounce techniques to reduce the export frequency
• Compare the stringified values and perform the export only when the value changes.

[94bryanr]

I am also interested in this. A few thoughts:

• Enabling the sqlite backend for wasm targets may be possible. See this PR to do so in rusqlite as an example.
• Could there be an addition to the current in-memory API so that you could optionally pass an impl of e.g. std::io::Write as an alternative location to write to? Then persisting to OPFS or localstorage would be relatively easy.

[davidhmays]

I would love browser based persistentence for local first applications.

I wonder if saving to ElectricSQL would be possible, enabling automatic sync to a back end as well?

[bselwayregis]

I've been using sqlite in the browser via the wasm release , backed by OPFS (as mentioned above) and its awesome

i imagine for cozo to accomplish similar, the easiest route would be like @94bryanr suggested and get a sqlite backend for the wasm build ? instead of managing/creating your own persistence layer in the browser

but yeah regardless, browser persistence would be AMAZING

i have some recursive traversals over json blobs in sqlite to crudely accomplish traversals and bounded graph queries, but would vastly prefer an engine fit to purpose

[zolero]

Any new findings or solutions so far? I'm quiet interested in this as well! Building SQLite alongside cozodb?

[creatorrr]

• Enabling the sqlite backend for wasm targets may be possible. See this PR to do so in rusqlite as an example.

That PR is close to being merged but only supports in-memory sqlite. For disk access, more work is needed to support a wasi target.

[creatorrr]

@chuanqisun @Volland @simonwh - have you guys tried integrating with IndexedDB? If not, it could be worthwhile implementing an indexeddb storage engine.

Relevant:

• https://github.com/cozodb/cozo/blob/main/cozo-core/src/storage/mem.rs
• https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
• https://github.com/devashishdxt/rexie

[creatorrr]

@zh217 thoughts on this? wasi-sqlite and indexeddb could be both promising directions

[Volland]

@zh217 also Libsql has a good wasm and in browser support . so it could be low hanging fruit

[Volland]

@creatorrr Unfortunetly we dont have a capasity to contribute to this feature