[2.0] observeSource API / docs confusing

[damonmaria]

I had assumed that observeSource would make the GroundDB collection match the collection it was observing. But that doesn't appear to be the case. I think I need to call keep first to make the collections match and then observeSource to keep it up to date, otherwise old records might be lying around in the GroundDB collection.

I can understand the need for both cases. Should there be a different method or some options to choose to leave records not matching the cursor?

Combining these two steps could also be more efficient since at the moment observeSource will run through all the existing docs again to do the initial added just after keep has. If there was a combined call then _suppress_initial could be used and the existing docs updated while calculating the ones to remove.

[raix]

what problem are you trying to solve? If you have data added by the user in offline mode you could do something like:

  const fooRemote = new Mongo.Collection('foo');
  const foo = new Ground.Collection('foo');
  foo.observeSource(fooRemote.find());
  Meteor.subscribe('foo', () => {
    // keep only current sub and data added offline
    foo.keep(fooRemote.find(), foo.find({ addedOffline: true }));
    foo.find({ addedOffline: true }).forEach((doc) => {
      // add the document to the remote db?
      fooRemote.insert(_.omit(doc, 'addedOffline'));
    });
  });

[damonmaria]

For this particular collection I'm just trying to keep a mirror offline. And that is what I had assumed observerSource would do. So at the moment I'm:

  const OrgProfiles = new Mongo.Collection('orgProfiles')
  const OrgProfilesMirror = new Ground.Collection('orgProfilesMirror')
  Tracker.autorun(() => {
    const subscription = Meteor.subscribe('orgProfiles.selectionList')
    if (subscription.ready()) {
      const orgProfilesCursor = OrgProfiles.find()
      OrgProfilesMirror.keep(orgProfilesCursor)
      OrgProfilesMirror.observeSource(orgProfilesCursor)
    }
  }

So wondering if there should be a method to do this (which could probably be more efficient that what I have here).

[raix]

  const OrgProfiles = new Mongo.Collection('orgProfiles')
  const OrgProfilesMirror = new Ground.Collection('orgProfilesMirror')
  OrgProfilesMirror.observeSource(orgProfiles.find())

  const subscription = Meteor.subscribe('orgProfiles.selectionList')

  Tracker.autorun((c) => {
    if (subscription.ready()) {
      c.stop();
      OrgProfilesMirror.keep(OrgProfiles.find())
    }
  }

[damonmaria]

Thanks. I see. And this is efficient in terms of access to the storage as keep only operates on what needs to be deleted.

To avoid opening another issue for a question. When observeSource calls setDocument it's EJSON.clone()ing it. So we end up with 2 copies in memory. Is the clone necessary? I would imagine for most that document isn't modified. Or could it be an option to not clone?

[raix]

Sure we could add an option eg. "ForceImmutable"?

[damonmaria]

That would be perfect.

[flippyhead]

We too are considering GroundDB for this use case.