About the project and persistence

riston commented 9 years ago

Hello, I am waking up the dragon (last commit two years ago). I am doing small research on mobile device data synchronization and seems this project has good abstract overview of using CRDT-s. The included examples seems to be working (simple and complex), but the question is about the persistence which is done by module kv both for client and server side. The syncing code is commented out and not working ? Also is this project dead or still could be used, what are the parts that needs to be rewritten to be fully functional ?

dominictarr commented 9 years ago

This project works and is in use. there have been no commits because it's finished. persistence was experimented with, but this was really useful as a lightweight in memory data.

how much data do you need to persist?

riston commented 9 years ago

Thank you for you answer, @dominictarr.

how much data do you need to persist?

Well it's not so much of the amount, currently all the data were saved in memory. I would like to get working practical example which saves data in client side(local storage should also be supported). Server side storing into some database (Redis support seems to exist) so it would be possible to scale more than one server instance.

What was the problematic part with the uncommented client side synchronization code ?https://github.com/dominictarr/crdt/blob/master/example/complex/client.js#L39

dominictarr commented 9 years ago

I probably just removed that because the example was sufficient without it. All you need to do is store the doc.history() data as local storage, and then stick it back into the client object again.

I made this to persist many scuttlebutts (which crdt is an instance of): https://github.com/dominictarr/level-scuttlebutt but it may be overkill if you only need a single instance.

It's actually pretty simple. Bascally, you just need to take the array you get out of crdt1.history() and store it, and then restore it by calling crdt2._update(data). So to clone a crdt instance the following should work:

var Doc = require('crdt').Doc
var crdt1 = Doc()
var crdt2 = Doc()

//insert data into crdt1

crdt1.history().forEach(crdt2._update.bind(crdt2))

basically that is what you have to do when you load from persistence, and to remember new updates, just have a listener on crdt.on('_update',..)

crdt.on('_update', function (data) {
  appendToStorage(data)
})

appendToStorage would be very simple on localstorage, and probably be an append stream of some kind on the server. This would be a good module and would work with any scuttlebutt subclass, I'd be happy to link to it from the scuttlebutt readme if you wrote it!

I think the only reason that I didn't already do this is because I was getting a bit carried away and trying to make everything a stream and 2) i never personally needed persistence on localStorage.

dominictarr / crdt

About the project and persistence #39