SQLSync is a collaborative offline-first wrapper around SQLite. It is designed to synchronize web application state between users, devices, and the edge.
With the rebase sync architecture, compacting the storage log is very easy and safe. At any point the coordinator can snapshot and start a new log from the snapshot.
The snapshot algorithm can work like this to improve efficiency on the clients:
coordinator snapshots the log at a particular LSN
appends a "end log" message to the previous log which contains the new log id and a hash of the snapshot
upon receiving this marker, a client can run the snapshot process locally and check that it gets the same snapshot
if it does, the client knows it can create a new local storage log from that snapshot and continue delta replicating from the server
if anything goes wrong, the client can drop their storage log and download the full snapshot from the server
This algorithm depends on the server continuing to serve the old log for some period of time - this time window determines how many clients will be able to efficiently switch over.
Since the coordinator also wants to be durable in a serverless setting, it's likely that the old logs will already be backed up to storage. Thus the coordinator can always serve from the last log lazily without keeping it in memory. (at least until GC runs and kills old logs, but that should never hit the immediately previous log)
With the rebase sync architecture, compacting the storage log is very easy and safe. At any point the coordinator can snapshot and start a new log from the snapshot.
The snapshot algorithm can work like this to improve efficiency on the clients:
This algorithm depends on the server continuing to serve the old log for some period of time - this time window determines how many clients will be able to efficiently switch over.
Since the coordinator also wants to be durable in a serverless setting, it's likely that the old logs will already be backed up to storage. Thus the coordinator can always serve from the last log lazily without keeping it in memory. (at least until GC runs and kills old logs, but that should never hit the immediately previous log)