SQLSync is a collaborative offline-first wrapper around SQLite. It is designed to synchronize web application state between users, devices, and the edge.
A more efficient replication system that only tracks page indexes rather than page contents in the log. The insight is that clients only need to fast forward to the latest snapshot. This can be computed by comparing the client and server versions and unioning all page changes in-between them. Then just serve pages from the current snapshot.
Note, that if concurrent writes are desired in this model, this does require some kind of actual snapshot functionality on the send side. To this extent, I've been researching copy-on-write systems to understand various ways to build this in a performant way.
Some rambling thoughts:
use the sqlite wal or build a similar wal of my own to buffer writes
consider that multiple clients can be requesting changes starting from different server versions, thus each needs their own changeset
although, one observation is that if you ignore truncation, changesets always either get smaller or stay the same size, they never grow
perhaps this means you could just build one snapshot from the oldest known client version and then send it out to all the clients with a tiny bit of filtering for more up to date clients
although, it's also desirable to stream out snapshots rather than building them in memory
currently it's nice that both the mutation timeline and the storage log share the same journal and replication abstractions, but this may be preventing me from doing things like easily truncating the timeline
need to be careful to maintain recovery after failure, currently since all writes go into a pending SparsePages object, any failure will result in the database safely rolling back to the last saved snapshot
Possibly makes #8 and #9 unneeded.
A more efficient replication system that only tracks page indexes rather than page contents in the log. The insight is that clients only need to fast forward to the latest snapshot. This can be computed by comparing the client and server versions and unioning all page changes in-between them. Then just serve pages from the current snapshot.
Note, that if concurrent writes are desired in this model, this does require some kind of actual snapshot functionality on the send side. To this extent, I've been researching copy-on-write systems to understand various ways to build this in a performant way.
Some rambling thoughts: