Replication over http

[opsb]

This weekend I want to start looking at replicating between orbitjs and a rails backend. I'm probably going to start simple and just send some json patches over websockets. This will only solve part of the replication problem though as using json patches alone isn't enough to know if the server or client side have diverged. This is more likely to happen on the server side e.g. a user's privileges might have increased so they need to do a full sync on any resources that have been added.

Having spent some time with pouchdb I've had a look at the replication protocol used by couchdb and while it solves the replication problem it's not particularly well suited to a rails backend (it uses global revisions which don't map well to a sql store).

At this stage I'm just going to be getting more familiar with websockets and how using json patches alone would work. In the longer term though I want to find a robust way to handle replication. Does anyone have any experience in this area or have any ideas about what a good approach might be?

[JannikStreek]

Isn't this already implemented (without websockets)? Given you have a bidirectional connector from your REST source to e.g. a localStorage source and you call find on the REST source, it will replicate the returned data to localStorage. If you then add/change data in localStorage it gets added to the REST source. To keep in sync with your REST source, you could call find in certain time intervals.

[opsb]

@JannikStreek Thanks for your input, it's very appreciated! That's certainly an approach I've considered, at least as a starting point, there's a few issues that I've been thinking about though:

1) I need a realtime solution - hence the websockets 2) Efficiency - my find calls are bringing back large object graphs, this is expensive and one of the reasons that I'm looking at orbitjs (serialization currently takes up to 10 seconds on the server). Using json patches rather than the json api structure will allow incremental updates which will be far more resource efficient (assuming the diffing algorithm server side is efficient, something else to think about...). 3) Relationships (with ember-orbit) - related to (2), because relationships are stored on both sides I need to pull down the objects on both sides so that their relationships remain in sync on the client (I think). 4) The jsonapi_source somehow needs to integrate local changes with the result from the find call, I need to see how/if this is currently handled.

I'm at a pretty early stage with this so any thoughts would be very welcome (especially if there's an existing way to do it that satisfies these requirements :) ).

[pixelhandler]

@opsb I started an implementation of orbit.js using a SocketSource in this branch of my blog app : https://github.com/pixelhandler/blog/blob/orbitjs-socket/client/app/lib/socket-source.js using this service: https://github.com/pixelhandler/blog/blob/orbitjs-socket/client/app/lib/socket-service.js and a couple initializers : https://github.com/pixelhandler/blog/blob/orbitjs-socket/client/app/app.js#L29-L54

I gave a talk on it, here are my notes: https://gist.github.com/pixelhandler/5389c72c82d26fae8fb1

I was able to demo orbit.js using a socket connected to a memory source but failed to also use a localStorage in the middle. I'll pick this up again in a week or so migrating to ember-orbit.

[opsb]

@pixelhandler thanks! that looks like a great place to start.

[ahacking]

@opsb. 10 seconds is a very long time for server processing. How much of this serialization is taken up by JSON encoding vs database overhead how large are your object graphs? The reason I ask is I implemented a rails JSON serialization library which in my testing and use is very performant and architected for large results. I examined all of the options and wasn't too impressed with the rendering overhead hence writing my own. Rendering 50 thousand contacts in my app is under a second per thread/cpu and for limited result sets of a few hundred objects per response is under 1 millisecond. You can find it at https://github.com/ahacking/to_json. My overall request turnaround in rails stack for 200-300 objects is typically 1ms-7ms per thread/cpu, I'm using rails-api controller with sequel, postgres and the sequel pg-streaming gem.

[opsb]

@ahacking yeah, I was pretty amazed at how long the processing took, the database overhead is only about 0.5 seconds (according to newrelic), the rest is all ruby.. I'll take a look at to_json and see if it can improve things a little :)

[ahacking]

@opsb I'll be keen to hear your feedback. Any questions or queries just raise an issue over there. At some point I want to issue a fully tested gem for wider consumption so your feedback will be very welcome.

[opsb]

@ahacking I'm on another project for the rest of the week now but I'll try and squeeze a quick tryout in.

P.S. I agree about RABL using an insane syntax!

[ahacking]

@opsb. No worries. I just added parallel benchmarks to the ToJson lib and I get 1.4 million independent serialize operations per second (serializing a single address and not aggregate which is even faster). That benchmark is running on my 4 core Intel Ivy bridge laptop. Scaling is linear when using separate ruby processes until you hit the I/O ceiling for disk or network.

[opsb]

@ahacking that does sound pretty damn impressive! I'll try and give it a go this week.

[JannikStreek]

@dgeb I would like to support JSONPatch on the server (to ease updating process on the client side), but to do so I would need to provide a date from the last update on the client side (because the server has no idea of the current state of the client). As far as I can see, this is not currently possible with Orbit, right? Or how do I provide such a date?

[opsb]

Stale now that the rethink branch has been merged in.