v2 usage - Githubissues

lorensr commented 8 years ago

Hi raix, I'm looking at your current grounddb-caching-2016 branch and hoping to come up with a recommendation for the Guide for offline data with Meteor 1.3.

Common use cases

Are these the two most common reasons why a dev might want to persist data?

Ability to load page while offline and still read data
Ability to change the data while offline and sync up when online
Ability to render a data-filled page sooner during an online pageload (instead of waiting for a subscription to be ready)

For 2, Meteor sends outstanding methods on reconnect, but this package no longer persists them across reloads. There is also no direct syncing of changes from the cached collection to the server collection on reconnect. You can call update on a Ground.Collection, but you'd have to either do your own syncing logic, or be building an app that didn't need syncing, eg an account-less todo list app that only saves data on your browser.

Offline reading

For 1 and 3, I think the easiest and most generally-applicable method would be for entire normal collections to just work / appear to have data. For 1, you'd use the appcache package and the below?

const Lists = new Mongo.Collection('Lists');
const ListsCache = new Ground.Collection('ListsCache');

// Cache Lists
ListsCache.observeSource(Lists.find()); 

Meteor.subscribe('lists.public')

Lists.find = function(...args) {
  return ListsCache.find(...args);
};

Lists.findOne = function(...args) {
  return ListsCache.findOne(...args);
};

// reload page, and Lists.find() returns data even though the Mongo.Collection 
// is empty, and doesn't hydrate

Jumpstart template rendering

And for 3, instead of Template.subscriptionsReady:

{{#if Template.subscriptionsReady}}
  {{> Template.dynamic template=main}}
{{else}}
  {{> App_loading}}
{{/if}}

https://guide.meteor.com/ui-ux.html#subscription-readiness

we'd need a different helper, like listsDataReady:

Template.foo.helpers({
  listsDataReady() {
    const cachedDataAvailable = !! Lists.findOne()
    return cachedDataAvailable || Template.subscriptionsReady()
  }
})

Cache size

When should we recommend trimming the cache?

At some point you'll hit quota, but there's not a good way to tell in advance (see this localForage issue). Perhaps we could listen for quota exceeded errors and call a user-provided a hook?

Ground.onQuotaExceeded(() => {
  ListsCache.keep(Lists.find());
});

Is hitting quota at all likely? How many moderate-size docs would it take to fill eg 10MB? (quotas are dynamic and across the board, but 10MB seems on the low end)

Are there other drawbacks to a large cache size? I'd guess for IndexedDB and SQL, queries wouldn't slow down much with data size like they do with minimongo?

The conservative approach would be trimming on subscription ready:

Meteor.subscribe('lists.public', () => {
  ListsCache.keep(Lists.find())
});

LDubya commented 8 years ago

Some people want to write Meteor apps that are purely offline and client-side. So without even a Meteor server component. Using the device as the storage medium. Localstorage has a ~10MB cap. Using "localforage" makes sense because then we can have IndexedDB by default.

raix commented 8 years ago

@lorensr It looks good - atm Ground DB II doesn't support outstanding method calls - this is probably something to add later on together with multiple tab support (both were in v1 but flaky due to the design of Meteor core packages and Ground db)

I'm thinking that it might be useful to add:

a package for backwards compatibility
helpers for general subscription ready states etc.
intelligent handling when hitting quotas

ilan-schemoul commented 8 years ago

Indeed @raix I confirm that I (and so other devs) need : 1) A reactive data source like .ready() for meteor subs to know when a subscription is ready (very important for me). mycollection.isLoaded doesn't seem reactive, am I right. 2) The ability (like in V1), in a seperate package if you want (tough I don't see the need), for Meteor.call() to be resumed on reconnection (pretty important also, it's sad that the V2 make all the app who use your package worst than before, isn't it ?) 3) Mutiple tab support (very important for sure, it's a Meteor principle ...). Wait, NO mutliple tab support does that mean that if I save offline with your package info in a tab user need to ... refresh ... the other tab. But it's horrible as users will think that it's the same when offline as online they will think that they'd lose every data when they change the tab offline. THIS ONE IS the most important I hope it will come very very soon :D Great work in any way @raix I just hope V2 could be used without destroying the user experience. Is there an ETA because if old Ground is deprecated and V2 is not usable I should drop support for offline until then.

raix commented 8 years ago

@NitroBAY I agree with you - the V2 does contain speed improvements (non blocking storage) atm. I'm not sure where MDG is heading with regard to DDP - it seems like Meteor is in flux which is sad imho. Anyway it's the main reason why ground db II is more decoupled from Meteor. (eg. no reactive helpers)

Determining whether or not a subscription is ready, GDB II is just a cache it has an "loaded" event that could trigger a "ready" reactive variable. Re subscriptions that's handled by regular Meteor - but Meteor publications are not tight to a specific collection, you could have multiple publications pushing data to the same collection, making a "collection" subscription ready flag difficult to generalize.

Using redux you have the option to store things offline - you might even write persisted actions allowing a later resume. (you can reason about this which makes debugging easier)

For tab sync it's mostly a matter of using the localstorage changed event to update all clients - and have them update their state accordingly.

But sure, you'd have to wire everything up yourself, V2 is just the result of preparing / having the option to leave the Meteor stack. You can rig other sources than DDP...

ilan-schemoul commented 8 years ago

@raix Hum it's interesting, as I'm pretty new (2/3 months) to Meteor I don't know much about low-level mechanisms as DDP, besides, I think, Meteor is voluntarily opaque with low-level mechanisms (I think they explicitly say that in doc). I may be dumb but I don't know how to use 'event' even after reading that https://github.com/GroundMeteor/db/blob/grounddb-caching-2016/EVENTS.md all I know about Meteor's event is Reactive data source with get and set. When I dive into a grounddb object I just see isLoaded true/false which doesn't seem reactive. So please can you share some code about how to use your event. Redux ? I don't know what is it. I taught you were just using localstorage/indexeddb (which I know since I learnt how to use Service Worker) and when Meteor.status().connected() you send the change over the wire to the servers. Do you think you'll re implement yourself tab sync at any moment or you try to keep ground db as simple as possible (I hope than yes but AS you already made such an amazing work for free it would be indecent to complain about your nice package) ? Why would I want to drop DDP, it is a part of the core of Meteor and it's great for me now ? By the way : on forums, and by reading your message, I see a lot of complains about Meteor and some people assume that 'Meteor failed', one says that 'Meteor is divided' etc. I never understood why as for me Meteor is just incredible and I accept the whole Meteor with Blaze, reactivity, DDP, the great idea of all calculate in client, the fact of share code between server/client. Tough I don't remember how I found Meteor (maybe Meteor found me) I'm sure that I fell in love mainly because of this idea of merge client/server, and calculate stuff on client.

raix commented 8 years ago

@NitroBAY Reading the events.md there are actually some reactive variables :)

Not sure whats going on in the forums - I've stopped following a while back.

I've been using Meteor for +3 years - they had alot of cool ideas - but the last year have felt like things got offrailed.

The CLI build times are ridiculously slow, blaze is being deprecated and it seems like ddp etc. is too. I've produced alot of packages, and hearing that atmosphere and the package system is being deprecated in favor of npm isn't helping - feel like I've waisted time.

The reason why I chose Meteor to begin with was to avoid having to build / glue my own stack... I'm not using Meteor on work we use a plain react/redux/webpack stack allowing us to use the tools out there - something Meteor doesn't quite do... yet? (hope theres still time for mdg to take leadership - wish them the best)

lorensr commented 8 years ago

On Mon, Aug 29, 2016 at 2:15 PM, Morten N.O. Nørgaard Henriksen < notifications@github.com> wrote:

The reason why I chose Meteor to begin with was to avoid having to build / glue my own stack... I'm not using Meteor on work we use a plain react/redux/webpack stack allowing us to use the tools out there - something Meteor doesn't quite do... yet? (hope theres still time for mdg to take leadership - wish them the best)

hey morten, i'm curious – what kind of tools doesn't meteor allow you to use? just webpack-based ones? there is a webpack atmo package, but i haven't tried it. you can use react and redux fine in meteor – you can use any npm lib on front or back, and you could use them all in lieu of meteor core libs, and just treat meteor as a nice build and deployment system that you don't have to write complicated config files for 😊

ilan-schemoul commented 8 years ago

I finally figure out the event system.

localUser.once('loaded', () => {
});

(which I don't really like, I mean event emitter with node.js is cool but if you want to make event driven development with Meteor you have to use reactive variable NOT event emitter imho...).

raix commented 8 years ago

@NitroBAY you could extend with a reactive ready or do a pr. something like eg.

import ReactiveVar from '@meteor/reactive-var';

 class NitroDB extends Ground.Collection {
  constructor(...args) {
    super(args...);
    this.ready = new ReactiveVar(false);
    this.once('loaded', () => this.ready.set(true));
  }
  isReady() {
    return this.ready.get();
  }
}

const db = new NitroDB('db');
Tracker.autorun(() => {
  console.log('ready?', db.isReady());
});

-

@lorensr Thanks, yeah I know about react in meteor and the apollostack is aiming for redux etc.

Re tooling, sure you have to write a bunch of configurations that's definitely a downside but at the same time most external tools today are also aware of these and use them.

I'm using:

microsoft code as my editor, it can't connect to the meteor node instance for debugging
wallaby so I get tests and code coverage while I type
plain karma and mocha tests for unit tests - plain simple, stable and fast (no extra layers)
HRM with webpack meaning I get fast page reloads - meteor has become slow

lorensr commented 8 years ago

Oh interesting, thanks ☺️

On Monday, August 29, 2016, Morten N.O. Nørgaard Henriksen < notifications@github.com> wrote:

@NitroBAY https://github.com/NitroBAY you could extend with a reactive ready or do a pr. something like eg.

import ReactiveVar from '@meteor/reactive-var';

class NitroDB extends Ground.Collection { constructor(...args) { super(args...); this.ready = new ReactiveVar(false); this.once('loaded', () => this.ready.set(true)); } isReady() { return this.ready.get(); } } const db = new NitroDB('db');Tracker.autorun(() => { console.log('ready?', db.isReady()); });

@lorensr https://github.com/lorensr Thanks, yeah I know about react in meteor and the apollostack is aiming for redux etc.

Re tooling, sure you have to write a bunch of configurations that's definitely a downside but at the same time most external tools today are also aware of these and use them.

I'm using:

microsoft code as my editor, it can't connect to the meteor node instance for debugging

wallaby so I get tests and code coverage while I type

plain karma and mocha tests for unit tests - plain simple, stable and fast (no extra layers)

HRM with webpack meaning I get fast page reloads - meteor has become slow

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GroundMeteor/db/issues/164#issuecomment-243325348, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPVmLb_2F9VUfDYGrM3YZMF0J2VrRPrks5qk6f7gaJpZM4IOTFa .

ilan-schemoul commented 8 years ago

@raix I used your code and yeah this is way better. I think this should be how you do it, instead of trying to create your own reactive system, but it's up to you.

import { ReactiveVar } from 'meteor/reactive-var';
import { Ground } from 'meteor/ground:db';

export class RGround extends Ground.Collection {
  constructor(...args) {
    super(...args);
    this.isReady = new ReactiveVar(false);
    this.once('loaded', () => this.isReady.set(true));
  }
}

Thanks for your support anymway.

raix commented 8 years ago

Im not using my own reactive system, event handling isnt my invention :)

ilan-schemoul commented 8 years ago

Why did you choose another event handler than Meteor's reactivity @raix ?

Meteor-Community-Packages / ground-db

v2 usage #164

Common use cases

Offline reading

Jumpstart template rendering

Cache size