Sync protocol improvements - Githubissues

splitgraph / seafowl

Analytical database for data-driven Web applications 🪶

https://seafowl.io

Apache License 2.0

386 stars 9 forks source link

Sync protocol improvements #530

Closed gruuya closed 1 week ago

gruuya commented 2 weeks ago

The changes in this PR involve:

new more robust replication protocol (formalized in the clade proto files).
batch compaction, meaning compressing changes from a single sync into a minimal form getting rid of temporary intermediate changes
idempotent CRUD support

TODOS:

Further segmentation of logic pieces that are not strictly needed in the sync writer
~A lot~ more unit testing, and a bit more integration testing