How to achieve offline support?

[paldepind]

Is it possible to use Apollo together with a client side database to achieve offline support? I found #307 where some comments imply that it should be possible but without explaining how.

Ideally we'd like to persist the data that Apollo fetches to a local database and use that database instead of fetching from a remote when the client is offline.

[stubailo]

You might be able to use something like this: https://github.com/rt2zz/redux-persist

The goal is that you can use some standard Redux offline library rather than needing something specifically for apollo, but I don't know if we have all of the necessary features for this. Can you list some of your requirements? I think all we need to do is write up a guide about offline use, and give a few examples, but I don't have a good understanding of what people need. For example:

1. How do you know if you're online and offline?
2. If you come online, do you want to refetch all data or keep using cached data?
3. What are the requirements for offline operation? What do you expect to happen with mutations, for example?

[paldepind]

Thank you for your response @stubailo.

redux-persist is great but I don't think it solves the problem that we have (I could be wrong though). One aspect is that you persist exactly what is in the store. That is fine as long as one is comfortable with keeping the entire persisted client site in memory. But for more complex needs I believe a proper client side database is needed (i.e. Realm, IndexedDB or Sqlite). This is necessary if you need to do efficient queries on the data (these would match queries supported by the GraphQL server).

We are building a mobile application with React Native and we need it to be functional offline. We would like to persist data that is fetched by Apollo. I.e. if a user looks at some fetched data it is persisted locally so that he can still access it if using the application offline later.

I don't understand Apollo very well at all. But what I imagined is something like client side resolvers similar to the server side resolvers in the GraphQL schema. These resolvers would be given the opportunity to retrieve requested data from the local database before the query is sent to the server.

We use React Native and check online/offline status with its NetInfo API. If the application comes online we ideally wan't to refetch all data that has changed since the user went offline. The GraphQL remote would of course have to support that as well. We'd also like to have all mutations that happened while offline sent to the remote. The mutations would also have to be saved persistently. A user might make changes offline, close the app, and then later go online and expect his changes to be sent to the server.

Of course handling online/offline applications is quite tricky. We don't expect Apollo to solve these problems but we're curious if we could actually implement what we want without Apollo getting in the way.

[stubailo]

Honestly it seems pretty hard to implement resolution of arbitrary GraphQL queries on the client. I guess you could reimplement your entire schema against the client-side database, but then you would be quite limited in what kinds of server-side features you could use.

Can you give me more specific examples of different views in your app, what data could be in there, and what queries you would need to display?

I think Apollo Client can give you some basic offline functionality out of the box, but it only knows how to run queries it's seen before. Since GraphQL arguments don't have any semantic meaning (there is no built-in sorting, filtering, etc) it would be hard to run totally new queries in a generic way.

[dahjelle]

Honestly it seems pretty hard to implement resolution of arbitrary GraphQL queries on the client. I guess you could reimplement your entire schema against the client-side database, but then you would be quite limited in what kinds of server-side features you could use.

(This is a bit off-topic from Apollo Client, but in the interest of discussing offline support…)

PouchDB is a client-side database that's nearly feature-equivalent to CouchDB on the server, and uses the CouchDB sync protocol to enable syncing between front- and back-ends. PouchDB can also act as a proxy to a CouchDB instance, so it wouldn't be unreasonable to write a bunch of GraphQL resolvers that would work on both Node.js and in the browser.

I expect some of the other client-side DBs could enable similar things, though, as far as I'm aware, PouchDB/CouchDB is the only one offering built-in sync mechanisms (with user-provided conflict resolution).

[stubailo]

I agree that this is possible - a similar concept is achieved by meteor with MongoDB and minimongo as well. I'm just saying that graphql is not the right server technology for this kind of database synchronization.

If you have some way of synchronizing the database to the client then you can definitely run GraphQL queries against that with Apollo Client.

[stubailo]

I'm just coming from the perspective where you are trying to query many different kinds of back end data sources and you can't sync them all into MongoDB or couchdb or similar.

[paldepind]

Honestly it seems pretty hard to implement resolution of arbitrary GraphQL queries on the client. I guess you could reimplement your entire schema against the client-side database, but then you would be quite limited in what kinds of server-side features you could use.

The goal is not to implement the entire capabilities of the server client side. Only some subset that would give sufficient offline features in our application. I.e. the local resolvers would be given the chance to get the data locally but they are allowed to fail if the user is trying to do something that only works online.

I am aware of PouchDB. It is a great solution to creating offline first applications in many cases (I don't know why you'd use it with Apollo though). But for various reasons we can't use it. We are using an SQL database on the server among other things.

I think it would be really great if the different ways of implementing offline support with Apollo could be explained or sketched out in the documentation.

[stubailo]

My current theory is that for at least some applications it will be sufficient to not have any resolvers on the client at all, and just use previously fetched query results stored in the Apollo Client cache (something that already works now out of the box).

Perhaps to start a deeper discussion we can talk about a case where that isn't enough? I think it will be a lot easier to think about specific needs rather than "offline" in general, which can be a very broad topic with lots of different tradeoffs.

[paldepind]

I think the theory sounds accurate. Is there a way to make the client cache persistent?

Maybe one case where that isn't sufficient is if your server supports querying a list of something over a range and you'd like to support that offline as well with different variables. If you've previously queried parts of or the entire list it should be possible to persist those results and perform the query on that data.

[mquandalle]

Perhaps to start a deeper discussion we can talk about a case where that isn't enough?

There is a lot of demand for a version of Wekan that would work offline, it should

• be able to persist boards and cards data across refresh (either in localstorage, IndexedDB, or AsyncStorage in the case of react-native)
• be able to resolve queries using this local cache
• be able to do some mutations (“add a card”) and have it reflected in the local cache
• keep a queue of unsynced mutations and send them to the server when an online connection resume
• update the client stores with the server data on resume

This seems quite ambitious in term of requirements, but actually the Meteor/DDP/Minimongo/GroundDB stack is already there, and apart from the conflict resolutions on the server-side, it’s almost easy to implement clients that works perfectly offline. Therefore the question is: should the same offline rich-client experience be possible in Apollo as well?

[smolinari]

This is a great discussion.

I was thinking, how about having a "syncing" query system built that isn't UI driven, but rather "offline system" driven, which can also fill Apollo's store (and automatically any browser persisted data system). In other words, data would be pre-fetched with a forward-looking algorithm, as if the user was working her way through the app. This basically would prime the store with supplemental data. Done smartly, it might even make any app using Apollo actually run much quicker. Possibly?

I do see there are still the disadvantages of syncing systems though. Don't get me wrong.... :smile: A lot more thought would need to be done on making sure all the readable data is accurate/ up-to-date.

For mutating, how about something like a "mutation stack", which stores any mutations made during offline operation. Once the device goes online again, the mutations in the mutation stack would be popped off (FIFO) and ran against the back-end. The results of the "proper" mutations would be final and updated in the store (and again, automatically in the browser persisted data system). An empty mutation stack with no errors, after the flush of all the mutations to the server is done, would mean the device is in sync.

One thing that might be ran into here is conflicts (a lot like with Git versioning). If someone changed data, while the other users was offline, it could be their own data is outdated. Boom.. Conflict.

There is still the big issue of syncing all of the data, as it might have changed on the server. This will probably need some sort of MVCC system, where the nodes and even objects must always have a "lastUpdated" or "version" property. If any data node has been updated since the last sync, any queries asking for those nodes or objects must be requeried. There would have to be a subsystem to do this version checking and updating, which isn't really part of GQL. It might also have to be ran, before the mutation stack is flushed....

Hmm.....

Just rambling.....I have no idea if it makes sense or is at all feasible or not. LOL!

Scott

[Akryum]

My current theory is that for at least some applications it will be sufficient to not have any resolvers on the client at all, and just use previously fetched query results stored in the Apollo Client cache (something that already works now out of the box).

Is there any documentation already available about how to do just that?

[paynecodes]

I think @Akryum's question is a good one. I understand that Apollo does this sort of thing in memory, and that is documented. I don't understand how the previously fetched queries could be persisted and retrieved between application restarts (think LocalStorage/AsyncStorage) by looking at the existing documentation. @stubailo, could you elaborate?

[tonyxiao]

Are there examples where people are using apollo in offline/online environments today?

[acamarata]

Does Apollo Client have any roadmap to have offline capability and features close to what current Meteor Client has via minimongo, etc.?

[stubailo]

Not at the moment, no - advanced offline support isn't a priority for us right now. Meteor and minimongo is a much better approach for applications which have sophisticated realtime requirements, but at the cost of relying on a specific database.

[abcd-ca]

Just wanted to add my 2¢ about why I need persistence in my app: My app is currently built in Meteor with Ionic Framework on mobile devices. My users have a job to do out in the forest where a data connection is often weak or non-existent. When online, they must sync to get the latest version of the collections they'll need, then they go out into the mountains and do a bunch of additional data collection. Then they return to a connected area and the data syncs. Offline persistence is critical so they don't lose their work in the event of a crash or by accidentally quitting the app. (I also have an app coming up with a similar use case except the users will be inside a building with a wifi-only iPad where the building doesn't have any wifi).

Meteor is especially nice the way it handles the online/offline situation. With a very weak connection, the connection is frequently lost and regained. Prior to meteor I would have had to monitor network connectivity, make a REST request to sync when connectivity was restored and manage retrying if it failed. Meteor just magically syncs when it can and my app doesn't die when the sync fails, it just catches up later. It really removes a big chunk of complexity. However, minimogo is volatile (non-persistent). I've had some luck with the ground:db package but it seems to be perpetually in alpha – and now people are migrating to Apollo so I think persistence with Apollo is salient – especially on mobile. Exciting times!

[bryanerayner]

Falcor has a really elegant model of caching that I believe could work really well with something like GraphQL / Apollo. The only problem is that using it currently requires 100% buy-in. If there were a way to only use JSONGraph (it provides good cache invalidation support, error support, etc. at the data storage level), and Falcor's Model, but using GraphQL for the network transport, you'd have a winning solution in my books.

[abcd-ca]

Already impressed with Apollo/GraphQL – I've started using it. Still very interested about this offline/re-sync topic

[fongandrew]

So, just to clarify, if all we need is persistence of previous cached queries, then we basically just use something like redux-persist on an existing Redux store that we then hook up the Apollo client to? That is, maybe something like this:

import {compose, combineReducers, applyMiddleware, createStore} from 'redux';
import {persistStore, autoRehydrate} from 'redux-persist';
import { ApolloClient, ApolloProvider } from 'react-apollo';

// Init Apollo client
const client = new ApolloClient({ /* your opts here */ });

// add `autoRehydrate` as an enhancer to your store (note: `autoRehydrate` is not a middleware)
const store = createStore(
  combineReducers({
    apollo: client.reducer(),
    /* Any other Redux reducers you have go here */
  }),
  {}, // initial state
  compose(
    applyMiddleware(...),
    autoRehydrate()
  )
);

// begin periodically persisting the store
persistStore(store);

Probably need to also add some logic to sanity check the persisted data as well, especially if you're changing schemas and whatnot.

Caveat: I'm currently just researching how feasible it would be to implement offline storage with Apollo -- I actually haven't tested the above code yet.

[smolinari]

@fongandrew - If you have success, do report here about it please! 😄

Scott

[funkyeah]

Congrats on the 1.0 for apollo client. Of course I'm particularly interested in the what's next section. Specifically "next-generation features like cache invalidation, persistent offline caching" which seems particularly relevant to this issue.

My use case is pretty much the same as @mquandalle in his comment, so I don't have a lot to add there, but I did want to point to a relatively new project takes a stab at addressing the primary needs I have but at the lower redux level https://github.com/jevakallio/redux-offline. I couldn't help but think when reading through the docs that it could help inform whatever solution ends up in Apollo.

[pi0]

@fongandrew Thanks for your starting point, here is a semi-working version.

Notes

• It fails loading from local cache sometimes on an unknown race-condition but we still have offline support
• During re-hydration process data.loading is not valid, we have to double check data contents is valid and if not valid still show loading
• Query fetchPolicy should be cache-and-network (not tested with other variants)

import {compose, combineReducers, applyMiddleware, createStore} from 'redux';
import {persistStore, autoRehydrate} from 'redux-persist';
import { ApolloClient, ApolloProvider } from 'react-apollo';
import {AsyncStorage} from 'react-native'

// Persist Apollo State
// https://github.com/apollographql/apollo-client/issues/424
// https://github.com/rt2zz/redux-persist
const store = createStore(
    combineReducers({
        apollo: apolloClient.reducer(),
    }),
    {/* Initial state */},
    compose(
        applyMiddleware(apolloClient.middleware()),
        autoRehydrate()
    )
);

// Begin periodically persisting the store
persistStore(store, {
    storage: AsyncStorage
});

[smkhalsa]

Any progress made on offline?

[sedubois]

See https://github.com/logux/logux-client/issues/20 and https://github.com/jevakallio/redux-offline/issues/7. Redux-offline seems stalled by lack of resources, Logux might be promising but quite a few open questions. (both projects were presented at React London)

Or need to build something custom...

[pi0]

As far as i know after great 1.0 release apollo Dev team has decided to separate storage layout so it can be shared across native and js platforms. I think it would be better to wait and somehow adopt with new upcoming approach. /CC @helfer @stubailo any small guide by you would be super useful :)

[helfer]

@pi0 yeah, we're working on it as fast as we can with the Airbnb team. We're going to publish our design docs here in the next few weeks so the community can participate and contribute as well!

We're a bit limited on bandwidth because I have other projects going on as well, but this is near the top of my todo list 🙂

[TheoMer]

What's the current status with this issue, as I'm keen to add offline support to my react/appollo/graphql app?

[kukagg]

Just to clarify the state of offline support, here is what I got to work:

Queries

At the moment apollo-client provides a full support for queries made offline with right options for fetchPolicy and cachePolicy. As apollo-client provides .reducer(), we can easily persist state with redux-persist so that’s a check.

Errors

Dispatched actions like APOLLO_QUERY_RESULT and APOLLO_QUERY_ERROR provide a nice way to control app state. As apollo-client emits errors when network requests fail one can catch them in your app reducer and dispatch an action to control isOffline state. Nice and easy way to displaying notifications like “No internet connection” or hiding UI elements that should be available online only. Worth noting that any errors thrown server-side in your graphql resolvers are caught too.

Mutations

This is something I’m still to solve as the app I’m working on requires a full support for offline create/update operations. I expect it to work with the help of “Optimistic UI” feature. We can instantly display the expected result as in pushing to the list or updating item’s content while try sending mutations in the background with a custom network interface. Appears to be a nice pattern for both UX and data persistence.

---

Is there something I’m missing? I’m not familiar with the codebase as some of the contributors. So the main question is how can we make this easier/more obvious?

[TheoMer]

@kuka Thanks. I left you a message on apollographql.slack, by the way.

[sabativi]

Hello @kuka,

Did you manage to handle network errors and how ? I am currently facing problems with this, but apparently I am not the only one https://github.com/apollographql/react-apollo/issues/758

For handling mutations, as redux-persist save everything to local storage and as we catch APOLLO_MUTATION_ERROR action which give us the id of the mutation, I was thinking on a simple solution that when connection is on again will read in AsyncStorage the queue of mutation_error and will send it back to the server.

What do you think ? did you manage to do it in an other way ?

[dk0r]

Here's a project (with a Medium article) that's approaching this topic: https://github.com/rauliyohmc/react-native-offline-utils

[manuelfink]

As @stubailo mentioned I guess apollo is not the same as an real data sync like minimongo and mongo offline db. However I think with queueing mutations something really similar can be achieved. I really think this would match 90% of the use cases without any problems.

I think the client just needs the following functionality to become fully offline:

• Queue Mutations
• Reply Mutations: manually triggered or use a (web) service running in background to automatically retry as soon as connection is available.
• Conflict resolution client: Allow a hook to be triggered if a mutation fails on the server.
• Queue State Indication: Are all mutations done or is there any queue

However there are some limitations and design patterns which must be respected, than this should work really well.

• UUID's: Do not use incremental ids, instead also use uuids in the sql world. this allows directly referencing objects and relations.
• Evaluation: Evaluate your data on client. Ideally use a universal data model, to ensure it validates both on client and server. Avoid conflicts on server.
• Optimistic Changes: You must always be able to predict the change which the server would make.
• Stateless Data Model: Your data should not relay on any server state.
• Writes! Avoid updates: As long as you only insert new records, where are no versioning conflicts! Modern databases like postgress, dynamo, ... can insert json sub-fields/records without overwriting the whole record.
• Versioning: Avoid Conflicts! if not avoidable, take care that conflicts can be resolved and merged in most cases. I believe in 95% of updates which address the same record, data can me merged without any manual conflict handling eg. then two different fields are changed. If versioning is really necessary, just allow mutations then the version number or timestamp of the previous version matches the one to be updated. If not force the user to get the current version and redo his changes. You need a bullet proof versioning strategy ahead of time!

[pcorey]

FYI @pi0 The recent 1.9.0 updates seem to have solved all of the loading issues related to Apollo queries and rehydration:

https://github.com/apollographql/apollo-client/blob/df42883c3245ba206ddd72a9cffd9a1522eee51c/CHANGELOG.md#v190-0

• Remove query tracking from the Redux store. Query status tracking is now handled outside of Redux in the QueryStore class. PR #1859
• Remove mutation tracking from the Redux store. Mutation status tracking is now handled outside of Redux in the MutationStore class. PR #1846

[pi0]

Thanks @pcorey for information 👍

[pcorey]

Hey all,

I've put together two articles that outlines my experiences with setting up offline queries and mutations with Apollo and Redux Offline.

Offline GraphQL Queries with Redux Offline and Apollo Offline GraphQL Mutations with Redux Offline and Apollo

Hopefully they'll help you out if you've stumbled across this thread.

[TheoMer]

Hey @pcorey

Many thanks for the break-down you've given.

Do you have a github repository where I can see the components mentioned in your article in action? I need to have a clearer understanding of your use of <Loader />, specified in your Rehydrated component.

[pcorey]

@TheoMer <Loader /> is just a generic loading component (in my case, I'm using SemanticUI). The details of that component aren't important. Anything of importance should be described in the article.

If you want a complete repo that outlines a lot of these ideas, check out ReggaePanda's redux-offline-examples project. It's a really helpful example project to get you going with Redux Offline and Apollo.

[tomsowerby]

Has anybody considered (or solved) how authentication could be managed? It may be the case that a user token has expired and they will need to log in again once online in order for the mutations to succeed.

[yuricamara]

Does anyone know a solution to use with Angular 4?

[PaulBrachmann]

Based on @pcorey's excellent articles, I came up with a prototype of a custom network interface & Redux store enhancer you can just plug into your app to basically get full offline-support out of the box (it utilizes Apollo's native caching & optimistic response features).

Here it is: Apollo-Offline

This solution is not framework-specific and should work with everything from a Vanilla JS client to using it with React (or even Angular).

Any thoughts, feedback, and contributions to the project are highly appreciated!

[TheoMer]

@MLPXBrachmann many thanks for this.

You've specified:

const Rehydrated = connect()

but have defined:

    <Rehydrate>
      <App />
    </Rehydrate>

Shouldn't that be <Rehydrated /> or am I overlooking something?

[PaulBrachmann]

@TheoMer you are absolutely correct! Indeed there is (now was) a typo in the README. Thank you for pointing it out!

[stale[bot]]

This issue has been automatically marked as stale becuase it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions to Apollo Client!

[stale[bot]]

This issue has been automatically closed because it has not had recent activity after being marked as stale. If you belive this issue is still a problem or should be reopened, please reopen it! Thank you for your contributions to Apollo Client!

[danieljvdm]

I think this issue has become relevant again since Apollo 2.0 has moved from Redux.

I was using redux-persist before to manage offline/saved state on my React Native app but now obviously can't. I have a working solution but I think it can probably be optimized a bit. Here's what I'm doing:

I created an "afterware" or "link" that simply forwards the operation on but also extracts and saves the cache.

const storeUpdatedLink = new ApolloLink((operation, forward) => {
  const newCache = cache.extract();
  setApolloCache(newCache);
  return forward(operation);
});

setApolloCache is a call to AsyncStorage in which I JSON.stringify the cache and save it.

In my App.js in componentWillMount where the redux persist call used to be (still is since I'm using redux for other stuff) I use a Promise.all on the redux persist rehydration call and the apollo client AsyncStorage fetch:

Promise.all([
  persistStorePromise(store, { storage: AsyncStorage, blacklist: ['nav'] }),
  getApolloCache(),
]).then((val) => {
  this.client = createApolloClient(val[1]);
  this.setState({ rehydrated: true });
});

And then simply:

render() {
    if (!this.state.rehydrated) {
      return null;
    }
    return (
      <ApolloProvider client={this.client}>
        <Provider store={store}>
          <MyApp />
        </Provider>
      </ApolloProvider>
    );
  }

This actually outperforms redux persist on app launch from my benchmarks but I'm worried about the link that I had to make to know when the cache updates. It seems a bit hacky and I think it's actually middleware rather than afterware as it seems to lag one request behind. Does anyone have any ideas of how to improve this, specifically the persisting aspect?

[2WheelCoder]

@danieljvdm I think your persisted cache is lagging one request behind because forward(operation) is called after you call cache.extract(). I haven't tested this yet, but this should solve it:

const storeUpdatedLink = new ApolloLink((operation, forward) => {
  const next = forward(operation);
  const newCache = cache.extract();
  setApolloCache(newCache);
  return next;
});

Thanks for taking a first stab at this. I think there is a lot of potential for some great offline patterns to build around Apollo 2.

EDIT

@danieljvdm Actually, I'm mistaken. The above is how redux middleware works, but because forward(operation) returns an observable, you need to wait for that observable to resolve before you extract the cache and persist it if you want to avoid missing your latest call.

const storeUpdatedLink = new ApolloLink((operation, forward) => {
  const observer = forward(operation);
  observer.subscribe({
    success: () => {
      const newCache = cache.extract();
      setApolloCache(newCache);
    }
  });
  return observer;
});

Or, something like that. When I implement myself I'll update. The docs on Stateless Links have some examples for using middleware similar to this.

[danieljvdm]

Since this issue is closed and there's a new opened one reference above I'll continue the discussion there.

[chmac]

@2WheelCoder / @danieljvdm I've tried a similarly naive approach to persist the cache which appears to have been working for at least the last 5 minutes!

const CACHE_KEY = "@APOLLO_OFFLINE_CACHE";

class ApolloOfflineCache extends InMemoryCache {
  constructor(...args) {
    super(...args);

    // After creation, read the cache back from AsyncStorage and restore it
    Promise.resolve(true)
      .then(() => AsyncStorage.getItem(CACHE_KEY))
      .then(cacheJson => {
        this.restore(JSON.parse(cacheJson));
      })
      .catch(error => {
        console.error("ApolloOfflineCache restore error", error);
      });
  }

  saveToAsyncStorage() {
    Promise.resolve(true)
      .then(() =>
        AsyncStorage.setItem(CACHE_KEY, JSON.stringify(this.extract()))
      )
      .catch(error => {
        console.error("ApolloOfflineCache.saveToAsyncStorage error", error);
      });
  }

  broadcastWatches() {
    super.broadcastWatches();

    this.saveToAsyncStorage();
  }
}

It's pretty ugly, relies on broadcastWatches() which is a private, internal method of apollo-cache-memory, but seems to be getting the job done, and doesn't rely on any timing hacks, etc.

I'm using ApolloOfflineCache in my client instead of InMemoryCache.

EDIT: Cleaned up the example a bit, removed flow types. This is all inside react-native, so I have import { AsyncStorage } from "react-native"; also.

[danieljvdm]

this seems pretty cool @chmac, do you have any more detail about how broadcastWatches is supposed to behave? Does it fire deterministically after store updates?