Request for Comments: Machine to Machine API for atomic data

[AlexMikhalev]

As a user, I would like to synchronize multiple atomic data servers - personal ones, one can be on the phone, another on raspberry pi and a third on a laptop.
I propose a single API endpoint: RPC sync (similar to couchd/couchbase), but data served in binary format, we can leverage serde to message pack serialisation or cap&proto + zstd compression to optimize the size of serialized binary going through the network.

For security we can work on: We can perform syncthing like adding devices to set mutual trust and certificates for device authorization and encryption.

Later this machine to machine sync can be the foundation of distributed/decentralized network.

[joepio]

Thoughts:

• It should probably work with Commits, as these are the fundamental units of change.
• The way CoachBase does this, is with a sync gateway, which is effectively one API for the User. Each user creates two channels: to and from. We could create a Tauri channel - we don't need HTTP. Normally in syncing, the annoying thing is authentication.

Use cases:

• Sync Atomic-Server on desktop with server.
• Keep multiple nodes in sync for backups.

Implementation ideas:

Use the /commits collection

It already contains all Commits. Replaying these, page for page, should give you all you need to rebuild the store.

Create a /sync Endpoint

Not sure how this should work, but we could always add custom query params and behavior on a new endpoint.

We could even have a fully custom implementation in Actix that uses websockets and sends binary blobs over the line. Not sure what it should contain, though.

Maybe:

• When calling /sync, a websocket opens. It sends a stream of Commits, starting from the first ever. Add a starts-from param if needed.

Adding a SUBSCRIBE_ALL action in WebSockets

We already have websocket support, including a bunch of commands. One of these is SUBSCRIBE ${subject} which is instructing the server to send alll Commits about a subject to a different machine.

We could add a SUBSCRIBE_ALL action, that sends all incoming commits.

If we combine this with quering the /commits collection, we could sync all things. But that process might be a bit error-prone.

Inspiration

• CouchCB /_changes

[joepio]

New idea: use iroh documents to sync two drives. Will elaborate later.