search

maxpert / marmot

A distributed SQLite replicator built on top of NATS
https://maxpert.github.io/marmot/
MIT License
1.86k stars 42 forks source link

CRDT #98

Closed gedw99 closed 10 months ago

gedw99 commented 10 months ago

Hey @maxpert

Would basic CRDT help with reconciliation ?

https://github.com/superfly/corrosion uses https://github.com/vlcn-io/cr-sqlite

Marmot could do this I think too I think. I need to see what cr-sqlite is doing

I am using https://github.com/ncruces/go-sqlite3 and looking into if I can use Marmot with it. Then I realised that a CRDT aspect would help with the Master Master reconciliation aspects of Marmot

maxpert commented 10 months ago

One of the things that cr-sqlite requires is enforces the client consuming SQLite to load the extension, without those extensions none of mutations work. But I see the difference algorithm they are using, I might want to have a implementation inspired by them for captured changes. I will dig deeper into it to see what can be done.

maxpert commented 10 months ago

Also quickly reading through, it also has same end state of last writer wins:

image

Which is exactly what Marmot provides today. So I don't see any value other than maybe the distributed counters or collaborative text (for which I will simply pick https://github.com/maxpert/sqlite-y-crdt)

gedw99 commented 10 months ago

Hey @maxpert

Glad I asked for your thoughts on this, because I also now remember the LWW ( Last write wins ) pattern that their CRDT design provides, and was not sure if Marmot offered the same.

Also really glad you brought up the https://github.com/maxpert/sqlite-y-crdt. https://github.com/yorkie-team/yorkie also does this and its in golang. https://github.com/yorkie-team/yorkie/blob/main/design/README.md is a pretty good explanation. They use Mongo DB for storage.
Might be useful resource if we want to add this feature latter to Marmot.

maxpert commented 10 months ago

So I did some deeper digging into cr-sqlite I actually made a quick python project and played with it with higher details. I think can inspire couple of ideas out of it, but I overall I feel like marmot should stick to its simplicity and easy to reason behavior. This will allow users to pick these libraries (and there are a lot like automerge, yjs etc.) to build on top.

gedw99 commented 10 months ago

Nice one. Yes I see what you mean about KISS ( keep it simple stupid ) principle.

think we should close this then....

gedw99 commented 2 months ago

https://github.com/automerge/automerge-go works well for auto merge too.

can send the sync messages via nats too.