Example of a REST/AJAX ISyncProtocol server

[ArtemGr]

Following up on https://github.com/ArtemGr/bounty/issues/28, if we want to synchronize the in-browser Dexie database with a server, we might implement the https://dexie.org/docs/Syncable/Dexie.Syncable.ISyncProtocol on the server for that.

I'd like to have a minimal NodeJS example implementing ISyncProtocol with a REST server. (Unless there is one already?)

Acceptance test woult be to show a working synchronization between a web page (such as a fork of https://observablehq.com/@theosch/path, which have the working Dexie code already on it) and a server running and/or available in the open, like with the repl.it, gitpod or runkit.

No need for persistance in the server side of the example, can keep the data in RAM.

--- Did you fix this issue? Go claim the **[$275 bounty](https://www.bountysource.com/issues/97626709-example-of-a-rest-ajax-isyncprotocol-server?utm_campaign=plugin&utm_content=tracker%2F191328815&utm_medium=issues&utm_source=github)** on [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F191328815&utm_medium=issues&utm_source=github).

[Speedwares]

Aha I think I see what you are going for. So the mutations and queries to Dexie now happen at the backend instead and reading the Isync protocol I can see that it is framework independent

[ArtemGr]

So the mutations and queries to Dexie now happen at the backend instead

I hope it's offline-first, so the mutations can happen anywhere (aka master-master), but I haven't looked deeper into that yet

[Speedwares]

So the mutations and queries to Dexie now happen at the backend instead

I hope it's offline-first, so the mutations can happen anywhere (aka master-master), but I haven't looked deeper into that yet

Ya I get what you mean saving in the browser before heading to the backend. Already spinned up repli server, will see and let you know if it works

[ArtemGr]

You can share your ongoing progress for the sake of collaboration, too! D

[Speedwares]

Ya makes sense

[Speedwares]

So I have been able to get a little version working I am using this https://github.com/nponiros/sync_server which writes the data to disk and shows it in a config.json file

[Speedwares]

Then I am using ajax to write to the server and that is working now

[Speedwares]

But all was done locally on my machine because I kept getting the no modules import error from observable and repll was having a no permission issue to write to disk. So I developed locally but can share the code if needed

[Speedwares]

Also, sync server does not work so well with node 14.0 so I am using node 12 for it but its currently saving the data successfully on the backend. I am trying to connect to indexdb right now and see if I can implement the offline first

[Speedwares]

I also did come across this https://github.com/nponiros/sync_client for syncing with the sync server but did not get much from the documentation and decided to go with sending it plainly instead of using the sync_client. I will hopefully have everything for testing soon. But will let you know when I figure out the saving to the frontend first before the backend.

[ArtemGr]

Have you looked at https://dexie.org/cloud/ ? I wonder if it's using the same ISyncProtocol? Maybe a more alive implementation than the older sync_client?

and decided to go with sending it plainly instead of using the sync_client

Sounds interesting. So you're implementing a synchronization plugin through the addons?

So I developed locally but can share the code if needed

Sounds good, maybe fork the bounty here and push the code into the fork. In the long run I'm not going to use the sync_server, too much unnecessary cruft there that will be getting in the way, but it will do for the proof-of-concept here.

p.s. Also if making this work is proving difficult then it might suggest that Dexie is not transparent/reusable enough. =

[Speedwares]

Have you looked at https://dexie.org/cloud/ ? I wonder if it's using the same ISyncProtocol? Maybe a more alive implementation than the older sync_client?

I have not actually. But saw it when going through. Ya I think it might also be using the same protocol but will check this out today.

So you're implementing a synchronization plugin through the addons?

Ya you might look at it in that way. The sync server did leave some examples on ways to send data to it so I used one of them to do this instead of using the client which was basically doing the same thing under the hood

Sounds good, maybe fork the bounty here and push the code into the fork. In the long run I'm not going to use the sync_server, too much unnecessary cruft there that will be getting in the way, but it will do for the proof-of-concept here.

Alright will do that.

p.s. Also if making this work is proving difficult then it might suggest that Dexie is not transparent/reusable enough. =

I won't say it actually is. Its just some of the packages are a bit outdated. Maybe the cloud might be what they are focusing on these days but will PR some code so you have a look soon.

[Speedwares]

Researched the Dexie Cloud and I don't think it uses the ISync Protocol. Looked well around and did not see them mention it anywhere. But I just PR the code for the client and the server.

I got the sync client to work very well with IndexDB and server. The sync is working properly now it sends an AJAX request every 2 seconds to make sure the two are synced.

[ArtemGr]

Thanks for the research, @Speedwares ! (Collaborative bounty paid out via Freelancer)

Definitely a signal that Dexie is not transparent/reusable enough. Because instead of reusing the two-way synchronization of Dexie, and/or reusing the Dexie addons, you have started your own implementation of the synchronization client.

Regardless, from your code I've got a bit of an idea about how the Dexie synchronization works. Seems like it's a typical "remember and send a set of changes" approach (previously mentioned)?

@TheOsch , any news on your end?

---

p.s. I have a simple dataset synchronization algorithm in mind which leaves the dataset tinkerable and scales optionally to merkle trees and then to cuckoo filters and such.

Going to start with the indexedDB version at https://observablehq.com/@speedwares/path and work from there.

Will be closing this issue and bounty soon if there is no updates.

[ArtemGr]

Google accidentally pointed me at https://blog.textile.io/local-first-threaddb/ cf. https://observablehq.com/@carsonfarmer/threaddb-usage-demo/2

[Speedwares]

Oh nice just scanned through the post and the observable notebook. You want us to try this out instead?

[ArtemGr]

Yep! Would be nice to have a fresh working example! (I also plan to read the paper soon)

[Speedwares]

OK sounds good, will see if I can replicate the example from the blog post. Do you want us to create an order on freelancer for it or you want to use bountysource

[ArtemGr]

Interestingly, the paper comes to what I call the JIT consensus: "conflict resolution is deferred to a point at which a decision is actually required .. means that apps can choose conflict resolution strategies specific to the tasks at hand". (Came to the same design decision about a year back).

And what I call owning the data they call "user-siloed".
(Seems like they have the right idea, but for simpler use cases their database is unnecessarily complex and less than tinkerable.
For a user to own the data they should be able to read and modify it, either by hand or with a third-party tool. Working with data through a simple code with minimal abstraction layer overhead is a step towards tinkerability as well. A YAML array is sufficiently simple and readable for that, ThreadDB - not so much).

I think I'll go with the tinkerable YAML storage and simple synchronization for now, and then it might be possible to implement a switch from YAML to a more complex database whenever the use case demands.

BTW, here's a real life example of creating a self-documented/commented YAML: https://github.com/ArtemGr/bounty/blob/8dfb276cf65bab495b00da27dc6ae5f64b88df67/data/firefox/firefox.js#L37-L46

OK sounds good, will see if I can replicate the example from the blog post.

I'm mostly interested to see it synchronizing with a specific server. And the steps you did on the server to make it work. You can use the ssh root@185.17.123.44 for that (can give access to collaborators here as well).

Do you want us to create an order on freelancer for it or you want to use bountysource

You can claim the bounty here or propose a Freelancer milestone (they even have an interface for that), whatever works for you.

[Speedwares]

And what I call owning the data they call "user-siloed". (Seems like they have the right idea, but for simpler use cases their database is unnecessarily complex and less than tinkerable. For a user to own the data they should be able to read and modify it, either by hand or with a third-party tool. Working with data through a simple code with minimal abstraction layer overhead is a step towards tinkerability as well. A YAML array is sufficiently simple and readable for that, ThreadDB - not so much).

Hmm this sounds interesting just checked ThreadDB out and just saw that it is built on top of IPFS which really I have been digging for a long time really an awesome protocol

I think I'll go with the tinkerable YAML storage and simple synchronization for now, and then it might be possible to implement a switch from YAML to a more complex database whenever the use case demands.

Alright noted this makes sense

BTW, here's a real life example of creating a self-documented/commented YAML: https://github.com/ArtemGr/bounty/blob/8dfb276cf65bab495b00da27dc6ae5f64b88df67/data/firefox/firefox.js#L37-L46

ok so the comm function adds a new line with the comment variable to the file, nice

I'm mostly interested to see it synchronizing with a specific server. And the steps you did on the server to make it work. You can use the ssh root@185.17.123.44 for that (can give access to collaborators here as well).

Will try this out. I have logged in previously

You can claim the bounty here or propose a Freelancer milestone (they even have an interface for that), whatever works for you.

Ok no problem, I think I will use freelancer and just track time there but will see if I can see the interface you are referring to