Add a MongoDB feature to add "redundant" information to the oplog (eg, change modifies to replaces)

[glasser]

The biggest performance issue with oplog tailing is that if you have a query of the form {complete: true, listId: 'foo'}, and you see an oplog update {$set: {complete: true}} on a document that didn't match the query before, you have to go specially ask Mongo to look up the document to see whether or not listId is true. This tends to mean that for some collections, every single update leads to every single observer fetching and re-evaluating. There is some level of de-duplication of fetching so most likely it only fetches once, but ideally we'd like to not fetch at all since "fetch mode" is the least efficient mode of the oplog observe driver.

One relatively simple way to improve this would be to add a feature to MongoDB to add "redundant" information to the oplog. Specifically, for the collections where you're seeing this sort of performance issue, Mongo would write "replace" operations instead of "update" operations. The observe driver would then have full context about the document and never need to do a fetch.

There's a tradeoff here: replace operations are larger than update operations, so doing this for all updates would cause extra replication bandwidth between servers and extra oplog bandwidth to oplog observers. But it would make Meteor's job much easier.

[mitar]

This might be resolved with #158.

[glasser]

Quite likely.

[mitar]

What I would also love is that I could do things like collection.insert({...}, {meta: ...}) and that content of meta would not be stored into a database, but would be only available inside observeChanges or something. So just passed through.

[glasser]

(I am confused though — have they actually implemented the change notification API?)

Yeah, some way to explicitly tag queries and modifications as related would be nice. My suggestion here is just because it's a relatively localized change to MongoDB (and a zero change to Meteor).

[mitar]

It seems it will land in 3.6. Example here: https://emptysqua.re/blog/driver-features-for-mongodb-3-6/