Generic CRDT library - Githubissues

pgte commented 6 years ago

Currently, the CRDT protocol being used in Peerpad is the native Y.js protocol, which is independent of IPFS. Each message is signed and encrypted, and then sent over the IPFS pubsub network for all participating nodes. In addition to that, individual p2p messages (Y.js requires individual p2p messages for bootstrapping) are channeled over libp2p connection using a specific protocol handler.

(The snapshot funcionality is independent of Y.js and uses the IPFS file API.)

Optimally, instead of using the Y.js CRDT protocol, we should put these changes into IPFS and make nodes follow / participate using a pubsub propagation mechanism (perhaps IPNS, if it's a good fit for multiple writers).

Some work towards this has already been accomplished by @haadcode's ipfs-log and tevere.

This issue aims to research and define the best way to implement a generic (Y.js-like) CRDT library over IPFS that is more native and does not have the need to have a specific protocol handler.

pgte commented 6 years ago

/cc @diasdavid @haadcode @vyzo @nicola

ghost commented 6 years ago

i used y.js 3 months ago and it was awesome. i mentioned to the authour of y.js that a golang version woudl be really useful and he said he wanted to do it. I guess he got side tracked. I also got side tracked in doing it.
Approaching him again is a good idea i think because i reckon he woudl love to see IPFS and his CRDT stuff integrated.

also a golang version is more useful than a JS because:

you can do the computations server side
you can now codegen the golang to WASM. WASM is new functionality in golang (is still on its own branch) and still a bit shacky, but for simple stuff like this it works i feel. But its much faster than JS in load up u time into the browser and almost as fast as native at runtime.
From JS you can wall the WASM and visa versa. From the golang WASM you can cal the DOM directly.

my 2 cents worth... :)

haadcode commented 6 years ago

@pgte let's get together and talk more about this as discussed last week. It's not a trivial topic as a whole :)

I had some time to finally properly work on A JavaScript library of CRDTs. I also realized that there's a strong need for one as the JS world's CRDTs are all over the place and not coherently presented in one place (code and docs). The module was created originally for OrbitDB and the G-Counter has been in use for a long time and I added G-Set, OR-Set, 2P-Set and LWW-Set and @shamb0t added PN-Counter. The library here is meant to provide the data structures as types / building blocks, so this is not specific to IPFS but I feel we'll need this.

Furthermore, I'd like to eventually include all known CRDTs in the library ^.

Generally, my current thinking is that we should try to keep the "pure" data types separate from causality tracking (clocks) and coordination and storage (IPFS) and create a layer on top. This, however, is not completely straight-forward (at least not in my head), so we should explore what the possible approaches are.

pgte commented 6 years ago

Published a generic CRDT library named peer-crdt that assumes immutable underlying store (like IPFS) and retrieval network, but is store and network-independent.

It implements a subset of the most famous CRDTs and allows you to extend this collection.

peer-crdt-ipfs implements the store and network for IPFS, using a Merkle-DAG to represent the operations and causal dependency.

ipfs / notes

Generic CRDT library #418