joepio
commented
2 years ago Thoughts:
- It should probably work with Commits, as these are the fundamental units of change.
- The way CoachBase does this, is with a sync gateway, which is effectively one API for the User. Each user creates two channels:
toandfrom. We could create a Tauri channel - we don't need HTTP. Normally in syncing, the annoying thing is authentication.
Use cases:
- Sync Atomic-Server on desktop with server.
- Keep multiple nodes in sync for backups.
Implementation ideas:
Use the /commits collection
It already contains all Commits. Replaying these, page for page, should give you all you need to rebuild the store.
Create a /sync Endpoint
Not sure how this should work, but we could always add custom query params and behavior on a new endpoint.
We could even have a fully custom implementation in Actix that uses websockets and sends binary blobs over the line. Not sure what it should contain, though.
Maybe:
- When calling
/sync, a websocket opens. It sends a stream of Commits, starting from the first ever. Add astarts-fromparam if needed.
Adding a SUBSCRIBE_ALL action in WebSockets
We already have websocket support, including a bunch of commands. One of these is SUBSCRIBE ${subject} which is instructing the server to send alll Commits about a subject to a different machine.
We could add a SUBSCRIBE_ALL action, that sends all incoming commits.
If we combine this with quering the /commits collection, we could sync all things. But that process might be a bit error-prone.
As a user, I would like to synchronize multiple atomic data servers - personal ones, one can be on the phone, another on raspberry pi and a third on a laptop.
I propose a single API endpoint: RPC sync (similar to couchd/couchbase), but data served in binary format, we can leverage serde to message pack serialisation or cap&proto + zstd compression to optimize the size of serialized binary going through the network.
For security we can work on: We can perform syncthing like adding devices to set mutual trust and certificates for device authorization and encryption.
Later this machine to machine sync can be the foundation of distributed/decentralized network.