ember-data issues since 2012. Maybe is there a way today? Integrating Push to Keep Client Data in Sync, Local Storage Support, Handling Bad Connections and Crashes: Gateway to Offline Support

[ghost]

I found this post from 2012 http://hawkins.io/2012/07/ember_wish_list/.

Since then something is changed, but not these points:

• Low Hanging Fruit - Integrating Push to Keep Client Data in Sync All complex applications allow data manipulation outside of user interaction. I mean not all data changes are done by users sitting on there devices. The server may be syncing something in the background which brings in new data. How is the client going to know about it? Is it periodically polling the server? That's crazy. It's easy to write code that pushes all changes. It should be equally easy to push that data into data store regardless of what push service you're using. Ember-data stores should define a simple interface to accept messages over push. Perhaps, the store itself is a web socket client which can be connected to your push stream. There are many different ways to do this. Everyone will solve this uncomplicated problem in the same way. This is exactly why the framework should do it. Ideally, there should be no problems if you switch from Websockets/Pusher/Faye/PubNub/Boxcar/etc. The messages always arrive.

• Low Hanging Fruit - Local Storage Support in Ember Data Loading screens blow. I downloaded the data. I shouldn't have to keep going to the server to get it again. You are probably using HTTP caching, but that still creates network requests. Getting good performance means cutting down on network requests. It's not very complicated to do in Backbone. Ember data should support local storage by default. For example, you create your store and you can set storedLocally: false if you don't want it stored. I think that adding local storage to support to the framework and enabling it by default would be a major win. Our app is dealing with a lot of data. Loading in data takes time and reduces performance. We only want to download that data once.

• Ambitious Addition - Handling Bad Connections and Crashes: Gateway to Offline Support Network connections go down. Browsers crash. These things happen. We have the tools to make applications failure resistant. Let's take your basic todo list application. The user adds a todo and for some reason the server is down. What happens? Do we just say "opps, sorry. Please try again." I don't think so. The framework itself can handle these cases. Here is a proposed solution. Use a messaging queue backed by local storage to buffer requests to the backend API. Requests that match a set of failure conditions (503, 504, or timeouts) are enqueued again and will be tried again later. This is a step towards offline support at the data layer. There has been some discussion about this. There are few things in the way. A request/operation object needs to introduced. This object is persisted in the queue. The adapter takes the request objects then does whatever logic is needed and sends them to the server. These objects would need to be tied to records as well. Since operations to individual records are being tracked, this means you can cancel operations to specific records. There are also race conditions. Hell, there are lot of complicated things to worry about, but solving this problem is massive. Imagine if applications simply got support for this by using Ember. Boom, your application has some level of fault tolerance and it may even save you a few customers. I'd say this is hardest problem to work on but the pay off is fantastic.

Maybe I'm wrong, but is there a way today to handle these problems?

[lolmaus]

I believe ServiceWorker is the solution to at least first two points. And it doesn't even require Ember Data to be modified.

[lolmaus]

As for the third point, is a much bigger problem than it may seem, mostly due to the necessity to deal with conflicts from concurrent edits from multiple users and even from multiple devices of a single user. Academic papers are being written to explore the matter and suggest possible solutions. I know two solutions based on such research: Conflict-free Replicated Data Types and Operational Transformation, each having multiple implementations.

Check out this awesome demo of OT at work. It exposes the complexity behind seemingly transparent conflict resolution.

Unfortunately, both CRDT and OT are virtually impossible to integrate into exisiting webapps. A webapp must be initially written with CRDT/OT in mind, otherwise your infrastructure will feel like two separate backends. This is because the CRDT/OT data persistence layer essentially eliminates the business logic layer (a backend framework) from your app infrastructure: the frontend directly communicates with the data layer. If you need to add some checks, routines, etc to user-data interaction, you'll have to employ data layer API hooks provided by a CRDT/OT framework (if any) instead of your cozy Rails/Phoenix/Symphony/etc pipeline.

And then there's Hoodie. Because it's based on CouchDB, it is extremely simple to use, and I believe it can be easily integrated with Ember Data (ED would communicate with the Hoodie frontend instead of the server). But because it's based on CouchDB, it only works for users in isolation. Once you need more than one user to interact with a data resource, you're in trouble. Hoodie does provide a way to do that, but it's an advanced, poorly documented path for the courageous (according to my findings a year ago).

There's also Orbit.js with decent Ember support, but I'm not familiar with it and dunno whether it provides any conflict resolution strategy. The readme and website don't mention "conflict" at all, though Orbit does look like an academic-grade solution to the problem. It does mention "client-first / serverless development", which might imply the same problem as Hoodie.

TL/DR: you can't make a frontend data layer offline-capable without tightly coupling it with a particular backend implementation.

[ghost]

If you need to add some checks, routines, etc to user-data interaction, you'll have to employ data layer API hooks provided by a CRDT/OT framework (if any) instead of your cozy Rails/Phoenix/Symphony/etc pipeline.

This is the point. CouchDB/PouchDB solution and Hoodie both miss this point. I need server check on every interaction. And I can also decide to throw an error if I want and stop all the pipeline.

But until today nothing in ember world can handle this situation.

Eg:

• OFFLINE LAYER (maybe IndexedDB) in browser.
• ONLINE LAYER (My API backend server)
• SYNC (when I want or on online/offline check), replay history of PUT, GET, PATCH, EVERYTHING in server and maybe server can throw(error).

Am I miss something?

[lolmaus]

Am I miss something?

The Ember Data team will never make it tightly coupled to a certain backend implementation.

But you can do your own implementation, have Ember Data talk to your IndexedDB layer. Ember Data doesn't need to be modified for that purpose, just customize your adapter & serializer.

[ghost]

Yes I know. I said Ember world. Not ember-data. I can talk with IndexedDB, what we today don't have is the synchronization mechanism for save in offline layer and replicate on the online layer.

This is what we really miss.

Something like redux-offline. But with Redux is a lot easier, I think, cause of reducer actions history.

[rwjblue]

We generally preserve this issue tracker for managing bugs in Ember Data. This doesn't really seem like a bug report to me, it seems instead like a feature request? I think that an issue in emberjs/rfcs would likely be a better place for this...