Intelligent subscriptions

[raix]

The idear is to have a simple way to keep as much relevant data on ground as possible - so the 5 or 10 Mb we have should be quoted and some algorithms should determine what data gets to live on ground.

This is developed as a separate package https://github.com/GroundMeteor/subscriptions

[raix]

There seem to be a way to simplify the code and get support for conflict resolution and better publish/subscription handling.

If done correctly all current code involving method resume will deprecate and give better room for subscription handling in iron:router. #3

[tagrudev]

So currently I am handling the iron router waitOn problem by checking the Meteor.status().connected otherwise I rely on the grounded data. Is this a good idea ?

[raix]

Not sure, I would rather depend on the data being in the collection or not - if not then wait. I'll talk to @cmather when I get there, but this is going to be solved - I depend on it myself

[nspangler]

@tagrudev I am doing the same thing. It doesn't necessarily seem like the cleanest implementation, but it works for the time being.

[isAlmogK]

Any news about this?

[raix]

I'm working on it - in the middle of a project were we need it :)

[isAlmogK]

cool, just sent you an email. Thanks

[isAlmogK]

@raix what's the current and correct approach to setup groundDB with iron-router waitOn I'm doing the following but feels a bit hackie

waitOn: function(){
    if(Meteor.status().connected){
      return [Meteor.subscribe('user')];
    }
  },
  data: function() {
    if(Meteor.status().connected) {
      return Meteor.user();
    }
  }  

[raix]

Wait on is an odd one - so when is the data subscription ready when using ground db... If the ground db is empty then its trivial to use the regular subscription - but if its not how do we know if the data is actually ready?

The data could be available in the offline cache - but it might not have been updated.

So we need the client to know about publish functions in some way - for now you could do something like:

  // Keep track of subscriptions
  var _groundHandles = {};

  var groundSubscription = function(name, publishFunction) {
    // Check the offline cache
    var cursor = publishFunction();
    // Fire up the subscription
    var handle = _groundHandles[name];
    if (!handle) {
      // If the subscription is not already started we initialize new handle
      handle = _groundHandles[name] = Meteor.subscribe(name);
    }
   // Check if we have offline data
   var gotOfflineData = !!(cursor && cursor.count());
    // Return some handle to iron-router
    return {
      ready: function() {
        // Lets rely on the offline data - if not found then rely on the subscription
        // handle
        return gotOfflineData || handle.ready();
      },
      stop: function() {
        // We don't actually want to unsubscribe the data when iron router wants to
        // We rely on intelligent subscriptions for this
        // so this is just a noop
      }
    };
  };

  waitOn: function(){
    return [groundSubscription('user', function() { return Meteor.users.find(); })];
  },
  data: function() {
    return Meteor.user();
  }

This is code is just written from the top of my head an completely untested - but its the principle

[raix]

• https://github.com/GroundMeteor/db#additional-api set the cleanup data to false - and do a manual clean up when needed - you might want to clear some collections offline cache on logout etc.

[raix]

the cleanup should be done when you got all the data you want offline - eg. data not updated by subscriptions would be removed. Note in the example we dont stop subscriptions - we might want to at some point.

The intelligent subscriptions is much more than this though,

[isAlmogK]

@raix thanks this is great I guess I need to test it out. Is the intelligent subscriptions at a phase for testing or early deployment would be glad to give it a test.

[raix]

its still being written - so I haven't actually run it my self yet - I'll let you know when baked from the oven :)

[isAlmogK]

@raix Is there a good way to abstract the groundSubscription function, seems a bit of overkill to add to each route?

[raix]

sorry - I just wrote it in one go - its a general function

[isAlmogK]

@raix I tested it the following example works great if you only have one subscription like

waitOn: function(){
    return [groundSubscription('user', function() { return Meteor.users.find(); })];
  },
  data: function() {
    return Meteor.user();
  }

But if you have more then one you start running into issues like so

waitOn: function(){
    if(Meteor.status().connected) {
      return [Meteor.subscribe('user'), Meteor.subscribe('organizations'), Meteor.subscribe('notifications')];
    }
  },
  data: function() {
    if(Meteor.status().connected) {
      return {
        organization: Organizations.findOne(),
        user: Meteor.users.findOne()
      };
    }
  },

Right it's somewhat of a pain, looking forward to getting this part of the package as plug and play.

[raix]

Have you tried using the groundsubscribe in the case of multiple subscriptions? (It's not clear to me Reading the code)

[isAlmogK]

Yea I trying it on multiple subscriptions, I'm debugging now. I am getting a weird error which might be due to something else.

"Exception in callback of async function: TypeError: Cannot read property 'count' of undefined"

Not finding my collection for the count

This is what I have

 waitOn: function(){
    return [groundSubscription('organizations', function() { return Organizations.findOne(); })];
  },
  organization: function() {
      return Organizations.findOne();
  },
  data: function() {
    var noOrganization = _.isEmpty(Meteor.user().organizationId) === true;
    return {
      organization: this.organization(),
      noOrganization: noOrganization
    };
  },

[raix]

Find not findOne... Have to return a cursor

[isAlmogK]

Yea that's stops the error but it's not returning any data stuck on the waitOn / sub

[isAlmogK]

It seems the main issue is with ground db not saving local data, it looks like the data is empty.

_storage.organizations.db.data [[],[0,[]],[1]]

[isAlmogK]

@raix yea that's the issue, it's clearing the local data for some reason. If I start with a different page that does not have the groundSubscription function I see the data but once I load a page with the function it clears the data. Any ideas why?

_storage.organizations.db.data [["r87GyCsjRnovobstm","address","streetAddress","Geulim 4/7","city","Rishon]]]

[raix]

https://github.com/GroundMeteor/db#additional-api set the cleanup data to false - and do a manual clean up when needed - you might want to clear some collections offline cache on logout etc.

[isAlmogK]

@raix yea that's the first thing I did, it had no affect

Organizations = new Mongo.Collection('organizations'); var groundOrganizations = new Ground.Collection(Organizations, {cleanupLocalData: false});

[isAlmogK]

Yea I tested it again it always clears the data, there is a possibility that cleanupLocalData has a bug

[isAlmogK]

I also tested out, still clears the data

Organizations = new Mongo.Collection('organizations'); Ground.Collection(Organizations, {cleanupLocalData: false});

[isAlmogK]

For anyone else this might help, the current way I'm working with waitOn is like so

waitOn: function(){
    if(Meteor.status().connected) {
      return [Meteor.subscribe('user'), Meteor.subscribe('organizations'), Meteor.subscribe('notifications')];
    } else {
      this.render();
    }
  },
  data: function() {
    return {
      organization: Organizations.findOne(),
      user: Meteor.users.findOne()
    };
  },

[tagrudev]

yes I am doing the same thing :) until the intelligent sub is on :+1:

[slowdownitsfine]

@almogdesign I'm experiencing the same problem with the grounded collection being cleared if I navigate away from the route and come back (even with the cleanupLocalData set to false). The grounded collection actually clears right after leaving the route.

I'am also making the subscription dependant on being connected, however I use the template instance helper to subscribe to it, not Iron Router.

The thing that strikes me as odd is that I'm grounding two collections: the Meteor.users collection and another collection I created. The users collection stays grounded (even while navigating away) while the other collection (its subscription depends on a session var) exhibits the clearing behaviour.

Any tips would be greatly appreciated, thanks :)

ps: @raix thank you for this very useful package.

[Anonyfox]

Any progress on this? @almogdesign's workaround doesn't work for me, with latest meteor/iron-router/ground:db .

[sferoze]

@dcusan I added meteor hacks subs manager package, and when I switch between the routes the subscription is still available using @almogdesign waitOn method.

Try adding this package and using it for the subscriptions on each route. It seems to still cache the data and even work offline.

[sferoze]

will intelligent subscriptions work with template level subscriptions?

I am using this pattern to load more items for the user: https://www.discovermeteor.com/blog/template-level-subscriptions/

I have set it up just like in the article. When the intelligent subscriptions is updated with ground:db will it work with this type of pattern? Any changes needs to the code?

[tafelito]

Hey @raix you still working on this? This is exactly what I was looking for, and I know you've been working ion this for a while, not sure what's the current status. Thanks!

[raix]

@tafelito to be honest, I'm holding back just yet - mainly due to the insecurity of where Meteor is heading ddp/apollostack

[tafelito]

I understand. Thanks @raix. So what would you recommend to do just for now? Can you think of any workaround around this? Maybe do the solution I saw out there just checking the meteor connection status when asking for the ready-ness status of the iron router subscriptions?