Caching / Offline - Githubissues

noris666 commented 7 years ago

Hello 4 all ! ;-)

We have a lot of questions about caching: https://github.com/inProgress-team/react-native-meteor/issues/153 https://github.com/inProgress-team/react-native-meteor/issues/150 https://github.com/inProgress-team/react-native-meteor/issues/147

I think it's time to do the improvement package. We need to determine which way we will implement data caching. @Mokto @grsabreu @spencercarli I would like to know your opinion ?

@grsabreu @tofrookie @elie222 Just good to hear specific cases that need to be cached? What are the tasks in your applications ?

My case: I have geolocations with data and need view information when i in offline. I don't have problems with scene routing and another technical aspects, i have one problem with caching data.

elie222 commented 7 years ago

SubsManager is about caching subscriptions. It won't help you when you're offline or when you restart the application. It will mean that if you go from screen A to screen B and then back again to A, you won't need to wait because you already subscribed. This also puts less load on the server a lot of the time. And you SubsManager will automatically unsubscribe after a certain period of inactivity. It shouldn't be too difficult to implement. I've started working on it just copying the original code for now.

noris666 commented 7 years ago

@elie222 ok, it is possible to share you implementation ? When you finish.

elie222 commented 7 years ago

Yes definitely. Nothing to share just yet. But hopefully I can work on it this week and get something working.

On 27 Oct 2016 1:41 p.m., "Georgiy Trokovets" notifications@github.com wrote:

@elie222 https://github.com/elie222 ok, it is possible to share you implementation ? When you finish.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inProgress-team/react-native-meteor/issues/154#issuecomment-256800279, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8oXy3LrYWGNF-huFgPQs8QybUWuyDkks5q4TacgaJpZM4Ki1kb .

tofrookie commented 7 years ago

realm is ok :)

fakenickels commented 7 years ago

I think this is totally needed to have, I think we can create some abstraction of minimongo collections in AsyncStorage API and use minimongo-cache on API to listen to changes in database and sync them to AsyncStorage. And just reinit the data in minimongo when Meteor.connect gets called and the offline. I can start this PR in the weekend if you agree with my idea!

noris666 commented 7 years ago

@grsabreu hey bro it is good way finally ! About AsyncStorage API you look redux-persist ? Better way mix our future abstraction with Redux.

fakenickels commented 7 years ago

No @noris666. I think that as not everyone are using Redux in their applications I think it can be done without any big changes in current projects using react-native-meteor. I think I can get to work on this idea when I get some free time.

fakenickels commented 7 years ago

"without any big changes" I mean to implement it as an internal feature of react-native-meteor

noris666 commented 7 years ago

@grsabreu ok i understand what you means, but if you need help write to me email when you will be free. My mail: georgiy@adaptive.kz ;-) Good luck ;-)

elie222 commented 7 years ago

If people want to try this out, here's my first attempt at converting the kadira subsmanager package to be used with RN: https://gist.github.com/elie222/516a716c0eb627039ed3fd7c3fe70983

And then you use it with something like:

const subsManager = new SubsManager({
  cacheLimit: 50,
  expireIn: 5,
})

subsManager.subscribe('posts') // this is instead of a Meteor.subscribe('posts')

fakenickels commented 7 years ago

@noris666 ok man! thank you

elie222 commented 7 years ago

I've removed it from my own code because I ran into a case where it wasn't working.

I feel like rewriting from scratch is probably easier because I don't fully get what is going on in the current subsmanager code.

It doesn't seem too complicated. We just need to set timers each time a subscription is made and at the end of the time to unsubscribe. I'm not sure how to handle a case where you're on a screen that uses a subscription. In such a case you'd want to wait till the person leaves the screen to unsubscribe I suspect.

We also need to reset timers when the same subscription is made again, but that doesn't seem too hard.

On 11 Nov 2016 03:35, "Gabriel Rubens" notifications@github.com wrote:

@noris666 https://github.com/noris666 ok man! thank you

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inProgress-team/react-native-meteor/issues/154#issuecomment-259958345, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8oX0Woc6GgftknnN9rfIJe8PCRc277ks5q9G8VgaJpZM4Ki1kb .

tofrookie commented 7 years ago

fightting:)

fakenickels commented 7 years ago

hey @noris666 I tried reaching you by email, have you received it? (Edit because of typo)

noris666 commented 7 years ago

@grsabreu yes, check me also by skype - funky_noise2009.

tofrookie commented 7 years ago

any update?

radiegtya commented 7 years ago

need update for this too.. without caching is just too weird for real world app

simonbelanger commented 7 years ago

Hey guys, I totally understand the pertinence of this feature, but I think it should be left out of this specific package... Imo this package should stay in tune with what's available in Meteor core. Maybe we can find a way to make the package more easily extendable by other packages?

radiegtya commented 7 years ago

miss the subsmanager package...

noris666 commented 7 years ago

@simonbelanger of course, now i also understand we need start another package to get that functionality. With this way we need organize team to start development process.

JulianKingman commented 7 years ago

I'm interested as well. What do you guys think of pushing data to redux where you can use a redux caching mechanism (like redux persist)?

fakenickels commented 7 years ago

I agree with @simonbelanger, I think a react-native-meteor "plugin package" is a good direction for this about Redux integration, as @JulianKingman is suggesting, I also thought about that in the beginning.

I think the main points of the persistent data feature are:

The current behavior: As subscriptions get stopped when the component gets unmounted and then the documents of the subscriptions get cleared up from the collection. How is that gonna be with caching?
Are we going to let the connection of the subscriptions opened the whole time? I'm not really into how bad this can be for mobile connections.

I think we can give a shot in the SubsManager implementation and commit the collections state in the AsyncStorage and the user will have to explicitly clean up the data with some API

JulianKingman commented 7 years ago

I'm working on a Redux integration, I should have it finished soon (next few days). Shall I submit it as a separate package or as a PR?

It's quite simple, it just watches for DDP changes and dispatches actions to a store, which it then returns, which you can access.

elie222 commented 7 years ago

Submit as a separate package.

On 30 Dec 2016 1:52 p.m., "JulianKingman" notifications@github.com wrote:

I'm working on a Redux integration, I should have it finished soon (next few days). Shall I submit it as a separate package or as a PR?

It's quite simple, it just watches for DDP changes and dispatches actions to a store, which it then returns, which you can access.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inProgress-team/react-native-meteor/issues/154#issuecomment-269836200, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8oX8X7gydFr-597sOc8x3oU0xntmxOks5rNZlVgaJpZM4Ki1kb .

noris666 commented 7 years ago

@JulianKingman yes i think better if it will be new package.

radiegtya commented 7 years ago

@JulianKingman : please give us info when ready, and can you share about how it can be used? Is it like caching on sub like this?

MeteorCache.subscribe('something');

thanks :)

adamgins commented 7 years ago

@JulianKingman really keen to see this.. I had just started trying to observe (actually just tthe subs to keep a redux store (using redux-persist, redux-thunk etc see https://medium.com/differential/building-offline-first-react-native-apps-b958acac0009#.aupqo284z ) up to date with Meteor collection info. Sounds like you may have this covered?

JulianKingman commented 7 years ago

@radiegtya I'm open to suggestions, but the way I'm currently handling it is like this:

import initMeteorStore from 'react-native-meteor-redux';
import _ from 'lodash';

// initiate the MeteorStore
const MeteorStore = initMeteorStore();
// now db updates will be saved to MeteorStore

// return all docs
MeteorStore.getState().collectionName;

// "findOne"
_.find(MeteorStore.getState().collectionName, {_id: "EgMN7J55bzKrkg7tPH0ApUGji"});

I think it's the easiest way, so I may stick to that for v1. I have it working, I'm just making sure I can get it to work with redux-persist without issue.

adamgins commented 7 years ago

@JulianKingman @radiegtya any chance I could try this out pls, could not find react-native-meteor-redux (i did see the slightly different spelling https://github.com/spencercarli/react-native-redux-meteor trying to do similar stuff)

JulianKingman commented 7 years ago

I uploaded it to NPM. Your mileage may vary, I'm currently having issues with redux-persist (see here: https://github.com/rt2zz/redux-persist/issues/244).

Edit: it's called react-native-meteor-redux, repo is here: https://www.npmjs.com/package/react-native-meteor-redux

JulianKingman commented 7 years ago

I fixed the redux persist issue.

JulianKingman commented 7 years ago

I created a function that makes it easier to call and updated the documentation on how to use it: https://github.com/DesignmanIO/react-native-meteor-redux

It doesn't currently take a find query. I'd be interested in creating a better API, but I'm not really sure how to do it. Ideally I think the package would export a single class MeteorStore that would wrap Meteor, so you could do MeteorStore('collection').find({some query}, {some options})`, but I'm not sure how to do that, class inheritance is still a bit of a mystery to me. If anyone is interested in helping or teaching me, let me know :)

radiegtya commented 7 years ago

Thanks for the great work @JulianKingman !!!

ASAP I'll try this..

wandonye commented 7 years ago

So cool. I have been searching for this for a while. Will try it soon

JulianKingman commented 7 years ago

Update: I implemented subscribeCached as the preferred way to access data. This integrates it with minimongo, so finds should work as normal. I found that returnCached was causing re-renders as it was switching data sources.

joaopiopedreira commented 7 years ago

guys, great work and good ideias here. I have a business Cordova/meteor app in prod for quite some time now and lost a lot of time developing my own offline logic (dealing with thousands of records, so I cannot rely on localstorage - 5MB limitation). I'm now porting it to react native and thinking on how to buil my offline store. because of my offline store volume, I have to find a way to write my collection data as json files to the device file system and then find an efficient way to load them into the collection objects (currently achieving this in cordova by using localforage and meteor map_Id api for fast loading ). Is anyone exploring something similar?

JulianKingman commented 7 years ago

Good question. Thousands is tricky, because so far my experience has been that getting and setting that many documents reactively is too slow. If you have a schema, you can use RealmDB (which is apparently fast), and then fetch data with method calls. It wouldn't be reactive, but it would be fast. Loading data into minimongo is not hard, you just use the getData method from this package, then do something like this: getData.db["dbname"].upsert([data array goes here]). You'll have to check if the collection exists and add it if it doesn't getData.db.addCollection(name).

joaopiopedreira commented 7 years ago

Hi @JulianKingman , thank you for your insights. I've been playing around with this for the last 2 days, but always come back to "ahhh, if I had full Meteor on the client...". That getData api has surely put me in the right track. I'm finishing now my offline wrapper in a proof-of-concept react native app. The app fetches some 2000 records from the server (via methods, not subscribe) and then stores them with AsyncStorage. I've managed to remove AsyncStorage limit of 6Mb on Android by editing the main.java file in the android build (as suggested by the folks at facebook). As you seem to know your way around react-native-meteor, would you mind taking a peek when it's ready? I'd love to have your opinion on this. Cheers :-)

JulianKingman commented 7 years ago

Sure, I'd be happy to. I'm curious what you need thousands of records for? There are some 'gotchas' with react native, but to me there are so many more positives than negatives.

joaopiopedreira commented 7 years ago

Agreed, love React Native. I have a cordova B2B app to manage field services (service engineers, home deliveries, that kind of stuff). Service engineers in particular need to have thousands of spare part references in the app (they never know which one they'll need for a certain job). My biggest customer has ~ 6000 records in this collection. When the app went to production, 2 years ago, it was not designed to be offline first. Had a bunch of problems because of that (users go into remote places or building basements, etc) and had to transform it into a 100% offline app. It works fine today and now I'm porting it to React Native - and loving the developer experience so far. But I'm so used to have all the Meteor goodies in the client and sometimes is so frustrating not to have them...

spencercarli commented 7 years ago

Hi all! I've been quite on this thread but have been following along. I spent some time thinking about it and wrote a blog post on a possible approach.

Much thanks to @JulianKingman as I referenced his package a lot when working on this - I just wanted to simplify the API a bit.

Would love to hear your thoughts so we can figure out how to unify these solutions and figure out the best possible option for users.

JulianKingman commented 7 years ago

I'm working on a realmDB-based approach that I think will nail the problem. First, I discovered the bottle-neck, which is this: AsyncStorage is a single key/value based storage. That means that if you store a collection, you serialize (stringify) an entire collection and save the whole thing every time you update it. If you have 10,000 documents, you're storing a huge array of objects that could be quite large, as a string, and diff0ing it each time a document is added or changed. There are a couple of solutions to this, one is to abstract out collections entirely and key the storage by _id (basically add the collection name as a field to a document). Then when rehydrating, you do some fancy lodash stuff to get things the way they're supposed to. Someone from redux-persist also suggested using debouncing and pausing (I didn't know you could pause), so it may be worth trying the persistor.pause() method. On the Realm side, Realm allows what it calls "Lists", which are basically child-records. This gives a bit more flexibility, makes things a bit clearer, and realm is supposed to be even faster than AsyncStorage, so that's cool.

joaopiopedreira commented 7 years ago

Hi @JulianKingman, as promised, here's my approach to solving this problem. I'm sharing here a repo forked from one of @spencercarli 's react-native courses (great course BTW). It shows a functional app working seamlessly onlin and offline. The data sync is done with methods. I'm loanding ~17k records in AsyncStorage. I also try to emulate meteor optimistic UI. Would love to have your feedback. Thanks. github repo

JulianKingman commented 7 years ago

How has it been working for you? You have some interesting things in there. I like how you use a class extending Collection from react-native-meteor for the offline collections, and am interested in how you are storing the results using the AsyncStorage API directly. Reading your code is helping me understand more of the Meteor API, too.

I have a couple of questions:

Doesn't offloading the diff-ing to the server create a lot of server overhead?
How long does it take to seed and store a 17k document collection? I think this has been the biggest bottleneck for me...
One problem I ran into is that dealing with the offline collection as a string and checking for changes took too much time. Have you had that problem? Maybe that's actually a benefit?
Overall I'm curious to know your thoughts about performance, and maybe more of your 'philosophy' about your approach.

I'm curious to see what we could take and generalize in a way that can be applied in more scenarios, and perhaps that's compatible with pub/sub? I think an ideal solution will 'just work', and not require specific server configurations, nor an unfamiliar API.

joaopiopedreira commented 7 years ago

Hi @JulianKingman:

The diffing in the server is very fast, no overhead at all. Remeber, in the server we have indexed collections so we can leverage that
It takes approx 9 secs to do the whole sync() process (including diffing in the server, diffing in the client and storing in AsyncStorage) with those 17K records
At the end of the sync() process, I just dump a collection.find() in AsyncStorage (I stringify it and store it as a string). So, one AsyncStorage key holds the full collection. I'll try and give you some more performance data related to this.
Performance wise, what I'm doing in that example I've published is to update the UI immediately and then wait for the collection sync() process to complete. If the server responds differently, than the results are reversed. This is an emulation of what the meteor folk call "optimistic UI".

joaopiopedreira commented 7 years ago

As a side note to complete the above, when I've mentioned that the sync() process would take 9 secs for 17K records, this is only for the first time it syncs; subsequent syncs take 2-3 secs.

In order to generalize this, since this approach has server side implications, we would have to make a meteor package (just for the server) and an npm package for the client. I'm still in the process of optimising the whole thing. Help wanted :)

you-fail-me commented 7 years ago

Are there any existing solutions to enqueue Meteor.calls while offline and execute when the connection is restored?

joaopiopedreira commented 7 years ago

@you-fail-me, you can do this:

import Meteor, { getData, call } from 'react-native-meteor';

getData().waitDdpConnected(()=>{
  call(`/offlineCollectionVersions/update`, {_id:offlineCollectionVersionsId}, modifier, (err) => {
    if(err) {
      return callback(err);
    }
    callback(null, offlineCollectionVersionsId);
  });
});

JulianKingman commented 7 years ago

So I've made some progress with realm, which you can try on the realm branch: https://github.com/DesignmanIO/react-native-meteor-redux#realm The caveat is that performance suffers a lot when the debugger is on, so you have to test without it.

I also came across this: https://github.com/smartdemocracy/react-native-local-mongodb @noris666 @Mokto @spencercarli could this be dropped in in place of https://github.com/petehunt/minimongo-cache? Then collections could be offline by default, and it appears to be quite performant.

didiercapozzi commented 7 years ago

Hello.

I would like to come back on this offline issue because it's the only step missing I would need in my application. Since March do we have any new solution for that? I've check the following link

https://medium.com/@miguelmorujao/offline-synced-database-with-meteor-react-native-and-realm-4a1da18de187

explaining the connection between react-native-meteor and realm but it's still a bit tricky in my case. @JulianKingman is https://github.com/DesignmanIO/react-native-meteor-offline also working with meteor call functions?

Thank you

inProgress-team / react-native-meteor

Caching / Offline #154