A good pattern for counting a query in a reactive way

[mitar]

Current common pattern for counting a query in a reactive way is to create an observe and then first count all documents in initial blocking stage, and then keep the count updated as changes happen (room counting example). This is problematic for large collections and large resultsets.

30 proposes some generic way to maybe address this, but there are maybe also some specialized ways for which I am opening this ticket.

[dandv]

It would help to mention how the pattern in publish-counts by @tmeasday, @dburles and @zoltan should be improve by core.

[tmeasday]

I think publish-counts is totally fine for smaller numbers (we use it for atmosphere's total # of packages, and I'm thinking that's starting to get too high).

For larger numbers, I've always thought there should be a way to do it using the low level oplog operations. You take a count(), then simply watch the oplog for i ops (or whatever they are called). There'll be edge cases around matching the timing of when you start reading the oplog and the time you took the count, but possibly that's not a big deal for most use cases. If not, then you might have to fetch the documents to be sure. Then you are stuck.

I've no idea how to do that but it seems like the tools should be there somewhere in the oplog code.

[mitar]

For larger numbers, I've always thought there should be a way to do it using the low level oplog operations.

Yes, but we (as end developers) should not have to parse oplog by ourselves, but should be able to use an existing API for that (like #30). Or maybe this should be provided as a special operation by core. Like observeCount. (I would also like to see observeExists, #46).

There'll be edge cases around matching the timing of when you start reading the oplog and the time you took the count, but possibly that's not a big deal for most use cases. If not, then you might have to fetch the documents to be sure. Then you are stuck.

Yes, in core that could probably be handled somehow. To match precisely the timing between initial .count() and oplog.