StorageAdapter & NetworkAdapter

[eberberich]

Hi,

what are the respective go part of the https://automerge.org/docs/repositories/storage and https://automerge.org/docs/repositories/networking/ adapters.

My goal is to create a doc in the browser (IndexDb) and then sync with a golang program, in there

• receiving the changes through- say WebSocket (or a similar adapter)
• writing the changes into a local file
• adding changes to the doc, that get then synced back to the browser

How to achieve this?

[ConradIrwin]

Hey @eberberich! There is no exact equivalent right now, though you could use automerge-go to build something like that. See the notes on syncing here: https://pkg.go.dev/github.com/automerge/automerge-go#hdr-Syncing_and_concurrency.

In short you'd use your existing network connection to run the sync protocol with the remote server; and periodically (or on every message) write the document somewhere you can find it again.

[eberberich]

Thanks for the quick answer ... I've read that before but some things are unclear:

• Are the SyncMessages compatible with the JS version?
• Do you have full documentation on the SyncProtocol - such that no handshake or other things are missed
• How the browser knows which data is missing in golang to be shared over this network connection?

And then there is the storage adapter. How would you get persistency (say on the file system) in Golang?

Thanks

[ConradIrwin]

Yes, the sync messages are compatible.

The protocol is documented here: https://automerge.org/automerge/automerge/sync/index.html. It's designed to encapsulate all the handshaking needed (including "which data is missing").

For persistency. If you're going to use a filesystem, you could do something as simple as "each time you get a sync message, apply it to the doc and write it out". There are more optimizations possible if you want to reduce the number of writes.

[eberberich]

Thanks! I'll check the details! Are there plans to add similar adapters for network and storage in Golang?

[ConradIrwin]

I am not planning to work on it soon, but if you wanted to work on it together we could.

[eberberich]

I would start with a simple file storage adapter ... I do not (yet) see an abstraction layer in the go code that eventually calls th functions load, save, remove, loadRange, removeRange as listed here - can you give a hint where to start from

[ConradIrwin]

Go doesn’t have indexedDb built in, so you’ll need to pick a storage format.

depending on the load you expect, and the level of concurrency you require, a file system based approach might work, or you could use SQLite or similar as a more featureful file system.

As the sync protocol runs, it sends changes over the wire which you can store as is, or immediately apply to the document. Immediately applying is a bit simpler to think about, but storing the changes as is gives you some space to trade off slightly slower reads for much faster writes

[gedw99]

Hey @ConradIrwin @eberberich

Glad this has been brought up.

https://github.com/3timeslazy/crdt-over-fs is a decent sync example. It's using S3 as the global store and has a CLI and Web GUI with an example todo being synced. it word work.

I am planning to adjust the Web GUI to use HTMX, because then its much simpler to use and the WASM can run 100% on the client and the server, and so its isomorphic and easier to extend, since its JUST HTML. See the Issue I made in the repo.

As far as Storage providers, https://github.com/tractordev/toolkit-go/tree/main/engine/fs has a load of storage adapters that are designed for sync. Its not using automerge, but I raised an issue there as its seems like a perfect match to me.

I am not saying any of this is perfect, but others are also working on similar efforts.

[gedw99]

immediately

Hey @ConradIrwin @eberberich

Glad this has been brought up.

https://github.com/3timeslazy/crdt-over-fs is a decent sync example. It's using S3 as the global store and has a CLI and Web GUI with an example todo being synced. it word work.

I am planning to adjust the Web GUI to use HTMX, because then its much simpler to use and the WASM can run 100% on the client and the server, and so its isomorphic and easier to extend, since its JUST HTML. See the Issue I made in the repo.

As far as Storage providers, https://github.com/tractordev/toolkit-go/tree/main/engine/fs has a load of storage adapters that are designed for sync. Its not using automerge, but I raised an issue there as its seems like a perfect match to me.

I am not saying any of this is perfect, but others are also working on similar efforts.

I am playing with NATS Jetstream teamed up with automarge-go.

This would allow sync to anywhere with full security backed in.

@ConradIrwin Is there a code element I can see where I can get the sync messages that run over the wire, so that I can stuff them into NATS. NATS will then publish that sync message automatically to the 1,000 of other users, and visa verse. NATS has topics and subjects and so it's very easy to map a file system / namespace into the system.

you can also do multi master SQL sync with NATS using CRDT. https://github.com/maxpert/marmot

[ConradIrwin]

@gedw99 This library doesn't do any networking, so you should probably do that in a wrapping layer.