Async/Remote Cursors for App Sync

[andrewluetgers]

So not sure if this would be better as a separate lib vs part of Immstruct but here's the idea. I'm implementing the "all your state in one place" pattern with my latest react project using Omniscient, great, love the pattern, using cursors etc. Then I think how do I do the network stuff?? I'm doing custom ajax off of a cursor but then I have to sync changes from the server, cool I can update state via web-sockets. Then I think, wait a sec, couldn't this work generically? So I think it could if there was some kind of remote cursor idea the server and client could stay in sync on a shared cursor location. If this could work all the server and client code could essentially ignore the fact that they are not on the same machine and just deal with shared cursors. In the client it would render state changes and push them to the server, on the server it would persist state changes and push them to the client. You might also want to persist on the client side for offline support, eagerly reading from and writing to local-storage optimistically while network activity takes place. If the network activity fails you can easily roll back to the old state!

So obviously this is a tall order, but I think it could be unbelievably awesome so I will be experimenting with this idea and wanted to get your general input on it. How feasible this is for naive last write wins remote syncing using shared cursors. Also where would I best start looking for implementing such a thing in Immstruct or if you even think this is the right place for it?

Thanks!

[andrewluetgers]

Have not used it but I think this is basically the same idea https://github.com/swannodette/om-sync

[mikaelbr]

I think, if done right, this could be quite nice for some projects (not all data models is a good fit).

Some concerns I have, though:

• Is your data structure transparent cross platform (client vs server)?
• Could this cause many updates with duplicated information? Only send patch requests (with diffs) through WebSockets?
• If the diffing is done, would there be a serverside aspect of this as well?
• How to handle batching? Or cancellation? (As a side-note: I've been thinking of adding a FRP/CSP layer as an extension for data and interaction layers for abstracting time in case of async operations for cursor-updates and handling things as cancellation/throttling etc.)

This could definitely be a cool idea if it works as expected, especially with undo/redo etc. I think this would have to reside outside of immstruct, just as a library pluggable on the top - not as a part of the core. I know @Dashed has done some work building on top of immstruct, so maybe he has some tips.

[ArnoBuschmann]

Does it help to reference the work of @elierotenberg here? https://blog.rotenberg.io/flux-over-the-wire-3/ Maybe things can be combined, somehow ;)

[andrewluetgers]

Yeah there are a lot of things to consider for sure.

As for data-structure across platforms I think it would feel odd to try to force your persistence layer to just take whatever state feels right for your app. I assume on the server side you would have some kind of persistence layer code that responds to events on the channel.

This could get messy, considering things like result sets that are paging and sorting will live in app state and obviously that is one way communication to the client. You would not want the client pushing back that change to server that originally came from the server. So you may want to differentiate what events need to be sent to the server and which do not. A dumb implementation would duplicate network activity but it would eventually settle down not unlike the angular digest loop, not the end of the world if it simplifies things dramatically.

One of the problems with this idea is that you could be storing a lot of state in memory on the server when you have lots of users, ideally you could get away with as little state on the server as possible pushing things out to a transaction queue is probably in order.

FRP/CSP feels like the right solution here and of course om-sync uses async transaction channels

I think the approach of @elierotenberg could be a good guide here, hypothetically you could just map changes on a channel to a flux store and I do think stores with actions and dispatchers may be a better pattern for sync vs a giant blob of state.

Another consideration is how you differentiate between state on the server that is shared across many users and that which is private. I think the flux over the wire is just doing shared-all. Ultimately access controls to determine who sees what would be needed.

[andrewluetgers]

Even if the flux model is not wanted https://github.com/elierotenberg/remutable seems a good starting place.

[elierotenberg]

hello :)

'private' state is nexus-flux/flux over the wire is implemented at the app level. I do this for users session. Client in the browser generate a private, crypto-secured token and sends a handshake packet to the server with this token. The server can then publish stores keyed with this token. Since the stores keys are not enumerable, knowledge of the existence of the key is bound to knowledge of the token.

Example: client generates secretToken='BFcRZAZcLqvX5UWL8BzqYn9P' and sends a Handshake action with payload secretToken='BFcRZAZcLqvX5UWL8BzqYn9P' to the server. The client also subscribes to the store '/sessions/BFcRZAZcLqvX5UWL8BzqYn9P'. The server can then update the store '/sessions/BFcRZAZcLqvX5UWL8BzqYn9P' knowing that only this client will know its existence. It also enables us to initiate session during server-side rendering or even share it between tabs/windows/devices if relevant.

[elierotenberg]

also, in my real-world apps, I have two separate Flux instances: a 'local' Flux (in the browser memory) and a 'remote' Flux (single source of truth on the server, components in the browser). Some actions in the local flux re-emits similar or closely related actions to the remote flux when appropriate.

[andrewluetgers]

@elierotenberg thanks for the clarification on privare state, thats a nice solution

[nstadigs]

I played around a bit a while ago with immstruct and scuttlebutts and it is definitely doable. Here's a very early, very wip, repo if anyone wants to work on these ideas: https://github.com/nbostrom/humdrum

[nstadigs]

I choose scuttlebutt for the communicative layer since it's built to be subclassed, is low level, is conflict free and because there are already a bunch of storage solutions built for it (like leveldb). https://github.com/dominictarr/scuttlebutt

[jeffbski]

@nbostrom mentions a few of these in his readme, but I will list here for easy reference

These modules facilitate creating and applying patches from Immutable.js objects which immstruct uses.

• https://github.com/zaim/immpatch Update immutable.js objects using JSON Patch operations, need to do .toJS() on diffs before using them since expects normal array and list.
• https://github.com/intelie/immutable-js-diff - generate diff patches between immutable.js data structures map, lists, sets
• https://github.com/intelie/immutable-js-patch update immutable.js objecs using rfc 6902 patches to maps, lists, sets - tested to work with patch generated from immutable-js-diff