support end-to-end tailable smart/capped collections

[jamesmanning]

tl;dr - tail -n 50 -f [collection]

The original SmartCollections already did the heavy lifting to do create/use a tailable cursor on a capped collection (the oplog). For scenarios where there are meteor apps surfacing logging information from a capped collection, it seems like it could be efficient/useful to support that end-to-end, such that the meteor server could publish this collection and clients could subscribe with a particular parameter for getting the last N items inserted into the collection along with the ongoing inserts that the server would publish.

The SmartCollections (and presumably the built-in Meteor.Collection in meteor 1.0) may be pretty close to that target efficiency already (being event-driven from the tail of the oplog) - the developer story I couldn't figure out is basically how to publish/subscribe the 'tail' of the collection as such.

The existing Meteor.Collection API seems to require (AFAICT) doing a sort by $natural descending, then a find with limit: N (say, 50), then fetching that, then reversing them in memory. Given the nature of the capped collection and tailable cursor, that seems much more hacky and less efficient than something that'd be able to do a tailable cursor for 'give me the last 50 documents inserted, then keep giving me new ones as they show up' (which AFAICT is viable with the tailable cursor?)

Related StackOverflow question:

http://stackoverflow.com/questions/19214889/how-best-to-tail-f-a-large-collection-in-mongo-through-meteor

Thanks!!

[arunoda]

You can do this with SmartCollection in another way. I'm not quite sure about the $natural and how it works with Meteor.Collection or Meteor.SmartCollections.

You can add a timestamp to the doc. Then create a query sort desc with timestamp and limit 50. Now when a new document added it will be received to the client and 51th doc will be removed. So now you've latest 50 collections all the time in the client.

This provided the same performance as tailable cursors. (since we are using oplog)

On Wed, Oct 9, 2013 at 1:14 AM, James Manning notifications@github.comwrote:

tl;dir - tail -n 50 -f [collection]

The original SmartCollections already did the heavy lifting to do create/use a tailable cursor on a capped collection (the oplog). For scenarios where there are meteor apps surfacing logging information from a capped collection, it seems like it could be efficient/useful to support that end-to-end, such that the meteor server could publish this collection and clients could subscribe with a particular parameter for getting the last N items inserted into the collection along with the ongoing inserts that the server would publish.

The SmartCollections (and presumably the built-in Meteor.Collection in meteor 1.0) may be pretty close to that target efficiency already (being event-driven from the tail of the oplog) - the developer story I couldn't figure out is basically how to publish/subscribe the 'tail' of the collection as such.

The existing Meteor.Collection API seems to require (AFAICT) doing a sort by $natural descending, then a find with limit: N (say, 50), then fetching that, then reversing them in memory. Given the nature of the capped collection and tailable cursor, that seems much more hacky less efficient than something that'd be able to do a tailable cursor for 'give me the last 50 documents inserted, then keep giving me new ones as they show up' (which AFAICT is viable with the tailable cursor?)

Related StackOverflow question:

http://stackoverflow.com/questions/19214889/how-best-to-tail-f-a-large-collection-in-mongo-through-meteor

Thanks!!

— Reply to this email directly or view it on GitHubhttps://github.com/arunoda/meteor-smart-collections/issues/37 .

Arunoda Susiripala

@arunoda http://twitter.com/arunoda http://gplus.to/arunodahttps://github.com/arunoda http://www.linkedin.com/in/arunoda

[jamesmanning]

@arunoda that's what we covered in the SO thread - the main problem with 'reverse and limit' is that the documents are in the wrong order in the initial set, and new documents are added at the front instead of the end. Since there's no support for 'last N documents from query' instead of 'first N documents from query', this seems to be a limitation of MongoDB I guess.

Is there an efficient way for the server side to reverse the collection that gets published out so it's back in natural/chronological order? Having to reverse the full array each time there's a new entry added seems inefficient AFAICT?

Thanks!

[bojan88]

@jamesmanning If I understand, you want the newest documents on top, right? With meteor Collection you can do this by sorting the collection: {$natural: -1} or {timestamp: -1} - you have to create timestamp field. But also you have to sort it on client too - {$natural: -1} or {timestamp: -1}

With Smart Collections it's a little bit different. You can publish with $natural sorting, but when it reaches the limit, it won't send new documents anymore. You have to use {timestamp: -1} for sorting on both client and server