Dev discussion: Model Syncing APIs

[owenb]

Goal

Achieve feature-parity with model support in Meteor (reactive templating and optional latency compensation) without forcing developers to use Mongo or a custom client-side lib.

Vision

SocketStream should not ship with support for models out of the box. Instead it should provide flexible, reliable APIs to allow other developers to write plugin modules (similar to Request Responders in 0.3).

Our APIs should allow developers to use any client-side framework (e.g. Angular, Ember, Backbone, or a custom roll-your-own) with any persistent DB (e.g. Redis, Mongo, etc).

Let's discuss what functionality we need to build to make this happen.

I have several ideas which i'll write up over the next few days.

Discuss :)

[yiwang]

Just start looking into http://sharejs.org/, is there possibility have an easy integration with OT as well for 0.4

[arxpoetica]

Maybe off topic, but also shouldn't force REDIS?

[owenb]

@yiwang I'd love to see third-party modules built on top of the SocketStream APIs we will provide which implement OT, but we're not going to do OT (or any other type of conflict resolution) in the SocketStream core

[owenb]

@americanyak Indeed. We don't force redis on you today (only recommend it), and won't in the future. The APIs I want to make will allow module developers to use any storage engine

[jcw]

The APIs I want to make will allow module developers to use any storage engine

Great. There are already so many decisions one has to take up front, so the more SS can stick to very loose and modular coupling, the better, IMO. Of all the choices involved, I consider SS to be one with the most "global" impact already, since it sits between such a wide range of other packages. I'm not even using express yet - starting out with the minimum has in fact helped a lot with learning how things tie into each other. And when everything is new and overwhelming, that really helps.

-jcw

[davisford]

Can we get more clarity on what exactly Meteor has/does that SS lacks. I'm trying to read their docs http://docs.meteor.com/#reactivity and it isn't entirely clear to me.

Are you trying to further abstract away the details of syncing realtime data? Currently, I find this to be fairly easy to do in SS, using well known pub/sub events. It sounds like Meteor is trying to further hide those details so you don't have to write code to publish or subscribe and everything is just magically wired up and syncs itself.

Some code snippet examples would be useful to understand the concepts SS 0.4 aims to strive for.

[pygy]

I don't think you'll manage to have something as clean as Meteor while being generic.

I know that at the very least, Angular will need custom logic to do it (you'll have to write a service, and add $watch calls where appropriate).

What you could define, though, is a data sync protocol, and have various adapters for the different client frameworks.

Other points:

• how do you handle horizontal scaling?
• how do you mange granularity and conflicts?

[kraz]

In my opinion the API must be as light as possible and exposing as mush as possible. That said if there are key points at the server's event flow, where i can hook up and intercept/modify/extend certain actions it will give me the opportunity to integrate my server modules with SocketStream. For performance reason it might be good point not to use server events, but callbacks. Or may be both - server events and server callbacks. Server events or callbacks can be attached at:

• Transport layer, where i can access the very beginning and ending of the requests and responds.
• View layer, where i can access the content and data (if there will be the possibility to attach default data into the view) before and after its being generated (if we assume that it will remain server side generated)

Exposing (transparently with web sockets) certain module functions to the client will make easy the view model synchronization.

In meteor modules there is possibility to embed assets other than code (stuff like - media, graphics). I personally, somehow, like the idea and if given module requires any assets it might be nice if they can be injected upon other client assets are being processed (compiled, combined, minified and gziped). Of course there must be on demand loading of assets, but its already being planned and implemented.

There is no need for default session store other than memory (if there is need for Redis, Mongo or what ever - there should be a separate module which can be easily included).

Talking about meteor - a week ago one of its creators wrote nice comment about modules/packages design.

[jcw]

Maybe it already exists, but if not or if no longer applicable for 0.4 - would it be an idea to hash out the pieces of SS 0.4 and come up with a list of everything in there, from a feature perspective? Something like:

• WebSockets - via engine-io, for pubsub and rpc between server and client(s)
• live-reload - reload client for client-side file changes, update in-place for CSS changes
• automatic compilation - CoffeeScript, Jade, Stylus, etc
• combine & minify for production (how does that work, btw?)
• plugin conventions for middleware
• path conventions for apps ("main", "iphone", etc)
• ...

I'm getting a lot going already with 0.3, but can't say that I fully understand how it all fits together. If SS had truly minimal example (so small that you never have to take anything out), plus a set of notes how various features can be activated or added on, then I think that could help lower the barriers to entry.

And while I'm writing down wishes anyway: an event "bus" with clear conventions of how to deal with server-side only, client-side only, and client-server events in both directions would be very handy, IMO.

[arxpoetica]

@kraz FYI there's sort of a debate going on right now over JS and shared server-client modules: https://twitter.com/domenic/status/299598843731902464 https://twitter.com/domenic/status/300373615604543488 https://twitter.com/paul_irish/status/300080151927062528

[kraz]

Thanks @americanyak. That is some neat information out there. :pushpin:

[owenb]

Hi guys

Thanks for all the above. I've read and digested it all.

@kraz I totally agree that high performance and exposing as much as possible is the most important thing.

I'm currently working on a new API for Websocket Services which builds upon the ideas on my latest 0.4 commit yesterday. I'll post a link to it here when it's done.

Cheers,

Owen

[owenb]

Ok guys the first 'draft' of my new idea is ready and integrated into SocketStream 0.4:

https://github.com/socketstream/realtime-service

The Realtime Service API is going to be a new open standard for creating plugin modules for realime frameworks, like SocketStream. All of the 0.4 code has been changed to use this, though there is still much work to do on the module spec (still need to add the ability to send code from the server to the client, etc).

The aim is for seamless integration from the user's point of view, but without any of the vendor/framework lock in of the past. An ambitious goal, but one I believe can be achieved with the right design, documentation and tests.

Comments and suggestions welcome.

[mgleason]

I think that link's no longer good - maybe the new link is: https://github.com/socketstream/realtime-service?

[mgleason]

AOT it looks great, although the API is TBD and I don't think I see anything specific to Model Syncing yet.

A quick note WRT "each service has it's own directory to store files (e.g. model definitions)": presumably the directory location will be configurable - unless you're assuming only static files will be stored there? I point that out only because many Node PAAS providers assume local storage is "ephemeral storage" that can disappear whenever your application stops.

[owenb]

Hi @mgleason Yes, that's the new link. I updated my original post.

Yeah the API is still a work in progress, but I'm happy with the basic design, so things won't change too much. I've been working on some model syncing stuff but have yet to commit anything to the public repo.

One of the things I still want to get right, before Realtime Services 1.0 is ready, is how we're going to handle offline stuff. I.e., should the Realtime Service layer help here, or should it be left for the service to implement.

Yes, you can specify which directory you want a service to store it's files in. These files will be part of your app - e.g model schema definitions, rpc methods, that sort of thing.

[mgleason]

So, I'm trying to get my head around how SocketStream is going to have feature parity with Meteor if it isn't going to ship with model support.

I think I'm seeing that SS will provide a minimal set of tools to facilitate syncing. And, hopefully, developers will create add-on's (similar to ss-backbone) that will actually do model syncing between client models and real databases (like Redis or Mongo). If that's the case, then the question is what features needed to make that easy. The Realtime Service is very flexible, but does it actually facilitate model sync? It helps by injecting assets and getting messages and error back and forth. It that enough? I guess I can't tell yet...

My understanding is that Meteor provides a local model store (MiniMongo) and a server store (MongoDB). MiniMongo is an in-memory store/db that watches for changes (events) to the local model. When a change occurs, it's pushed up to the server. On the server-side, MongoDB is polled regularly and when a change is found, it's pushed down to the client. This is how the models are kept in-sync. Note that the server (MongoDB) is treated as the master though so if there's a discrepancy on the client (MiniMongo) it's over-written by the values on the server.

MiniMongo is in-memory so it provides: 1) "Latency compensation" because changes made locally to MiniMongo are reflected in the UI immediately and then pushed up to the server. MongoDB is treated as master though so if there's a discrepancy between the local model in MiniMongo and the server model in MongoDB then the server pushes the current model down to MiniMongo and the UI get's updated to reflect the server's value. 2) If connection is lost with the server, then MiniMongo continues to cache local changes until connection is re-established with the server (unless the user closes the browser, then the changes are lost). 3) But, because MiniMongo is in-memory, it does NOT provide storage between browser sessions.

If offline means something like HTML5 local storage, here's a nice list of libraries which do this and provide fall-backs when the browser doesn't support it - https://github.com/bebraw/jswiki/wiki/Storage-libraries. It would be great if Realtime Services can add value in handling offline. Otherwise, one of the existing libraries might help.

Geoff Schmidt gave a talk a while back about DDP (Dynamic Data Protocol). He talks about it some in YouTube (at about 34:00 http://www.youtube.com/watch?v=34_oKFx43O4 - and here's a blog post introducing it http://meteor.com/blog/2012/03/21/introducing-ddp. And this guy actually created a client based on it https://github.com/sonyarouje/DDPClient.NET.

TL/DR - Maybe what's needed is: 1) Something like DDP would save developers from having to invent a protocol and set of messages to keep the client and server models in sync? 2) A message queue which will hold those messages, ensure delivery and resend/replay them (for when communication is lost or the client or server is overloaded/slow, etc.)

[mgleason]

Hopefully I'm not saying something stupid. But I also want to point out something - since it actually confused me for a while when I first found SocketStream and saw some comments about Redis and ss-Backbone and ss-angular that gave me the impression people thought it provided parity with Meteor:

• Meteor doesn't just provide Model Synchronization.
• Meteor actually provides Model Synchronization AND PERSISTENCE.

In fact, changes are written to MongoDB first and then synchronized back to the client. Whether its done in a way that will scale is another discussion.

[arxpoetica]

Just adding to this thread without reading everything above, but I didn't want to lose sight of the fact that someone's worked already on this problem a bit: https://github.com/korbin/socketstream-0.4/tree/master/mods/rts-model

from @korbin's own mouth:

"Havent touched it in a long time, but, this created mirrored model objects across the wire. I need to take a look at it again. It was...overkill." (Don't shoot the messenger!)

[paulbjensen]

For Dashku, I used a combination of a custom client-side model library, called bucket:

https://github.com/Anephenix/dashku/blob/master/client/code/libs/bucket.coffee

And I hooked into that with pub/sub, so that model updates got pushed from server to clients.