Use MongoDB change notifications instead of oplog with MongoDB 3.6

[mitar]

MongoDB 3.6 will contain an official change notifications API. Using that to observe changes should be much better than current oplog approach. We should probably implement a third (besides oplog and polling) driver to use change notifications.

[fermuch]

The changes API uses the aggregate pipeline¹, so it might be possible to have reactive aggregate queries in publications? That would be a huge plus. Just imagine being able to use $graphLookup² and have it update automagically!

[niallobrien]

Are MDG even interested in this any more? Seems that Apollo has taken priority.

[abernix]

@niallobrien Apollo and Meteor are separate projects with separate developers and there is no reason that Meteor wouldn't be interested in such a feature. That being said, our resources are not unlimited and help would be appreciated in looking into this.

[msavin]

Does "resources are not unlimited" === "we will continue to have one developer staffed on the Meteor project"?

[niallobrien]

@abernix I never mentioned Meteor, I specifically mentioned MDG and if I'm not mistaken, MDG develop both Meteor and Apollo...

[fermuch]

As most of the best additions for meteor were implemented first as packages, I think this feature would also be best implemented first as a package, and then, when it becomes stable enough, could be integrated to the meteor core (or not).

The meteor core needs to maintain stability and backwards compatibility (meaning this driver would only be added when mongo 3.6 is the minimum version supported by meteor, and all other older versions are deprecated). As an example, the oplog watcher was first implemented as a package, by kadira I think. After some time, meteor adopted it. Before it, we had long-polling for everything. When Meteor adopted this driver, backwards-compatibility was ensured (no need to change your code, but if you want to use it just add MONGO_OPLOG_URL to your env).

The same is happening right now with redis-oplog. It started as a proof-of-concept, then moved very quickly in 0.x versions, and now is stabilizing and mostly fixing corner-cases.

---

MongoDB 3.6 isn't even released yet (AFAIK), and there is already work being done to free meteor from mongodb for those who don't want to use it (after all, this was one of the most requested features since meteor was conceived). In a few releases more, mongo will be totally optional (and this is a good thing!), so creating a package for this new mongo API makes a lot of sense, or at least it does in my conception.

EDIT: I'm not affiliated with the MDG; everything I say here is based on my observations I've seen meteor had in the past.

[mitar]

meaning this driver would only be added when mongo 3.6 is the minimum version supported by meteor, and all other older versions are deprecated

Not true. Meteor already has support both for oplog and long polling. This would be just one extra approach and if Meteor is running on 3.6 it could be used, otherwise it could degrade to oplog.

[fermuch]

meaning this driver would only be added when mongo 3.6 is the minimum version supported by meteor, and all other older versions are deprecated

Not true. Meteor already has support both for oplog and long polling. This would be just one extra approach and if Meteor is running on 3.6 it could be used, otherwise it could degrade to oplog.

As far as I've seen, this approach would need you to write a query to the aggregation pipeline¹, which is currently not supported by meteor (you need to access the raw collection to use it) and is not compatible with the oplog driver.

How would a backwards-compatible driver work?

I can imagine adding aggregation support to the long-polling driver, but I can't see how it would work with the oplog.

[mitar]

I think you are confusing things here. This is a completely new API which also supports using aggregation pipeline API as a way to select which notifications you care about. The point is that you do not have to "oplog observe" the "notification aggregation pipeline" to get notifications. You simply use MongoDB driver to say "hey, inform me of any notifications on this collection, and apply those aggregation pipeline operations on top of notification documents before that". So, in the same way as oplog driver currently connects to the MongoDB oplog, change notifications driver would connect to change notifications. The fact that the latter is using internally aggregation pipeline syntax does not matter.

So this is orthogonal question to a question of support of aggregation pipelines in Meteor.

[msavin]

If there is one person who can pull this off while making it look trivial, it's @theodorDiaconu. He already has extensive knowledge of how LiveQuery works.

[hwillson]

Just a quick reminder to all here - we can't really even start to think about supporting Mongo 3.6 until https://github.com/meteor/meteor-feature-requests/issues/152 has been addressed. There's quite a bit of work involved in dropping Meteor's 32-bit support, but it has to happen before we can jump to Mongo >= 3.4. So while Mongo 3.6's change notifications API definitely looks interesting, and will likely be awesome with Meteor, there is still a lot of less exciting work that needs to be accomplished to get Meteor ready for newer Mongo versions.

[msavin]

What do you folks think Meteor developers could expect if this functionality were integrated? It looks to like it can be a cleaner way to consume changes compared to oplog - we can observe specific events instead of all of them - but it looks like it would still have the same issue with overloading servers. Perhaps the redis-oplog is still the right way to go about this.

[fermuch]

@msavin As far as I understand, the overload comes from processing each and every message in the OPLOG, instead of only what's really needed. With the changes notifications, only messages for currently active subscriptions would be watched, so why would it be the same?

In my mind, this would happen:

• User A subscribes to itemsInStock on Server B
• Server B creates a changes consumer and watches for User A's updates, notifying her if an item is added/removed/updated.

So when User A disconnects or stops the subscription, the watcher for Server B on itemsInStock can be deleted.

[raphaelarias]

@msavin

I've tried the redis-oplog, even thought I like it there was two bugs:

• Not considerting SimpleSchema options (I fixed)
• Some mutations, even thought processed in the server were not being pushed to the client.

So I still prefer to use a MongoDB-only solution.

The problem with overloading the server as far as I understand is:

• We have to read every single change.
• Every instance has to do the same

With the notifications API I think we can listen to changes for the things we care (Publications cursors) and it could be implemented in a way that only the server that needs it receives the changes (I think by default this would be the way to go).

To make it perfect, we could add a way to manipulate the notifications API when creating the cursor.

With these, we will have no overloading (more than what you need) and full control of reactivity, even disabling it (which will be great to still use publications but do not track changes in the documents nor update them).

[raphaelarias]

By the way, I would like to help in the efforts to bring the notifications API to Meteor.

[theodorDiaconu]

@raphaelarias do you have a bug submitted for Some mutations, even thought processed in the server were not being pushed to the client. ?

Now let's go back to Notifications API: https://dzone.com/articles/new-driver-features-for-mongodb-36#notification-api

It's a very nice API and it will solve SOME of the big problems (bandwidth , meteor cpu usage). But the fundamental issue is the fact that control of reactivity is limited.

It still does not allow what RedisOplog can do:

• Ability to trigger reactive events without persisting data (like the example of when someone is typing in a message and you want to tell that to the other party)
• Ability to make mutations that should not trigger reactivity
• Merge incoming changes into a single message (When you have 100 subscribers on the same collection on the same instance, you will receive 100 change notifications)

In terms of scalability, using a middleware like Redis is the way to go.

[raphaelarias]

@theodorDiaconu Yes, but I haven't investigated further to help understand what happened, as I'm not using RedisOplog anymore.

The thing is, to use RedisOplog add another point of failure (another DB and another package that you may not be able to maintain), so if notifications API can replace it, I'm on board.

• Ability to trigger reactive events without persisting data (like the example of when someone is typing in a message and you want to tell that to the other party) It does not solve, and as far as I understand it may be difficult to change how meteor works to natively support it. But with less bandwidth and CPU usage this may be less of a problem, isn't it?

• Ability to make mutations that should not trigger reactivity If, when implemented, we can pass options, or control the notifications API, logic dictates that we can deactivate reactivy.

• Merge incoming changes into a single message (When you have 100 subscribers on the same collection on the same instance, you will receive 100 change notifications) Doesn't Meteor reuse subscriptions to optimise the pub/sub? If that's the case and we can apply (using notifications API) filters and mutations to it, done. So we may not receive 100 change notifications.

[msavin]

I just found the documentation - its being called "Change Streams"

@fermuch Nice - it looks like that can work:

Change streams provide a way to watch changes to documents in a collection.

However, I do wonder if this is intended to target a collection or individual groups of documents. If the latter, and it could power LiveQuery, I would probably use Meteor forever.

@raphaelarias I'm on the same boat in regards to vendor support. If Redis-Oplog were a commercial offering, I might have more faith in it, but looking at what happened with most of the third party things here, it seems like a risky bet.

@theodorDiaconu I also think Redis has a place in Meteor, in fact, much of Redis marketing demonstrates how good of a combination it is with MongoDB. However, I am not sold that RedisOplog would be as reliable as LiveQuery is today. (Reliable means many things here.)

Maybe the right solution is to have a Redis package that works independently to MongoDB, and integrates well with Meteor's pub/sub system.

[ramezrafla]

If Redis-Oplog were a commercial offering, I might have more faith in it, but looking at what happened with most of the third party things here, it seems like a risky bet.

I second that too. Especially that redisOplog is a big package. When something breaks in a small package (e.g. oauth packages) we can fix in no time. But this is a complex beast!

@theodorDiaconu we had discussed this in the past. I really think you should look into a commercial license for redisOplog (what's $99 / year for a production app!). Maybe a short survey on the Meteor forum to see who is interested in it could be a starting point.

PS: Sorry for hijacking this post

[raphaelarias]

I would probably pay.

On 5 Oct 2017, at 4:49 pm, Ramez Rafla notifications@github.com<mailto:notifications@github.com> wrote:

If Redis-Oplog were a commercial offering, I might have more faith in it, but looking at what happened with most of the third party things here, it seems like a risky bet.

I second that too. Especially that redisOplog is a big package. When something breaks in a small package (e.g. oauth packages) we can fix in no time. But this is a complex beast!

@theodorDiaconuhttps://github.com/theodordiaconu we had discussed this in the past. I really think you should look into a commercial license for redisOplog (what's $99 / year for a production app!). Maybe a short survey on the Meteor forum to see who is interested in it could be a starting point.

PS: Sorry for hijacking this post

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/meteor/meteor-feature-requests/issues/158#issuecomment-334365717, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AExrJn0ro3oQn01Kr8u3s38Uq6WESphlks5spG3ggaJpZM4Ohy_i.

[dnish]

I guess adding an option where Meteor "ignores" changes within the database is really needed when the Mongo API gets added to the core. In our example case we've removed about 100k chat messages per day to clean our database. This killed our Meteor app with enabled oplog. We've moved to redis-oplog and can now control which updates are processed by Meteor and which not.

[theodorDiaconu]

Thanks @ramezrafla and @msavin for the ideas. Don't worry I won't let RedisOplog die. Will start investing time again in it and Grapher.

@raphaelarias The thing is, to use RedisOplog add another point of failure (another DB and another package that you may not be able to maintain), so if notifications API can replace it, I'm on board.

There are certain key concepts here:

• Redis Oplog is fully BC so adding it/removing it is done via a simple configuration
• As you scale you cannot avoid splitting in your app in microservices and using different type of databases that satisfy different needs: Searching, Big Data Analytics. Hence, adding "point of failures". RedisOplog was never intended for 100-200 online users.

[msavin]

+1 for paid package - especially if that can pave the way for more features like performance monitoring and so on

@theodorDiaconu can you please clarify what you mean with "RedisOplog was never intended for 100-200 online users"

[dnish]

@msavin I guess he means that this package was not made for "small apps", because they won't need it. I would also love to pay for it if it gets maintained. This really saved one of our chat applications which generates revenue, so it would be fair to "pay you back".

[mitar]

@theodorDiaconu maybe create an open collective: http://opencollective.com/ And then companies can be backers.

[msavin]

I'd be willing to help brand it, build a site, etc. Check out this branding idea :D

https://forums.meteor.com/t/meteor-scaling-redis-oplog-status-prod-ready/30855/263

I actually started some work on it but didn't want to be too pushy 😄

I know you want to keep it open source, but you can do paid open source with really lenient licensing. "The package is yours forever, but if you use it you have to send money to X." I believe that is much healthier for both sides.

You can take a look at BotPress as one example. They are open source but if you go above X operations or whatever you are expected to pay for it.

[msavin]

BUT - to bring it back to the original discussion, maybe the right solution is to use Redis for simple things that need to be really fast - like indicating that someone is typing during a chat - and using MongoDB + Change Streams for the rest.

I've noticed that MongoDB doesn't always write reliably - or it may take a while - so using Change Streams feels more reliable. If I understand correctly, RedisOplog assumes that the write happened right away, and I could see that causing some discrepancies, especially in collaborative apps.

[theodorDiaconu]

@msavin it is not intended for small apps, you won't benefit too much from it's performance.

@mitar I will do that, create a roadmap, and get a bit of budget.

Ok guys, I will start this project to bring redis-oplog to it's deserved form, @msavin I've sent you an email with my Skype id, so we can brainstorm a bit the "branding" part.

I think there can be a way to use ChangeStreams and RedisOplog together. I'll be waiting for their driver to mature to even begin thinking about such merge.

On topic: Their driver needs to be very smart to do the "merging" of streams. So that if you have 100 listeners for a change (with diff fields) on a collection, to receive only one message with all the info you need from the db, instead of 100. Oplog is doing this in the sense that tailing the oplog is done via one channel.

[raphaelarias]

@theodorDiaconu I agree. But the way it is right now I cannot trust the app and run my SaaS on it. It's a very sensitive, important and intricate area to leave a package run by itself without any kind of ongoing development and/or support.

We don't use very advanced things regarding pub/sub, and still, it was not a drop-in replacement, as I pointed out before.

In any way. I would use a paid version, but I think we should use Change Streams API as much as possible, to keep it simple. And then more advanced features or complements we use Redis.

So, if you can merge both, awesome. To do that Meteor needs to have a first-class (ideally) support for notifications API, otherwise it will have to hack it's way into it.

TL;DR: I would love a fully-supported redis-oplog and an awesome, open and flexible Change Streams API for Mongo. That's why I would like to help to bring this to life.

[theodorDiaconu]

Change Streams would bring absolute performance to our scaling problem, one use case:

Given a publication for collection "users" that contains some filters, limit and optionally skip options. In order to make sure that the image on the client is the correct one we have to listen to all updates inserts and removes on "users" collection. Given the scenario that "users" collection changes a lot, the watcher from Redis is still going to be bombarded. (Let's call this a FLS query)

Now, given Change Streams, and watching a FLS query at DB level, the only changes that come to the callback are the ones for that cursor only. Which solves a very important problem that even RedisOplog doesn't currently solve completely. It requires additional use of channels and namespaces, which works great, but ChangeStreams would solve it better.

Why RedisOplog and ChangeStreams are two different beasts ? With RedisOplog, changes are sent out to Redis. With ChangeStreams, there's no need to send changes anywhere. This is why the two, can't be properly merged together.

RedisOplog Advantages

1. Redis is a beast in what it does, it's very performant, with ChangeStreams, the CPU of the database will increase by a lot. But the DB can scale, so this is not a decisive factor.
2. Synthetic Mutations (emulate changes that aren't saved in the database)
3. "Silent" Mutations (that do not trigger reactivity)

ChangeStreams Advantages

1. Reactivity at the database level means one less point of failure
2. Solves the problem of FLS queries
3. Easy to implement, given the changes processor is handled at Mongo level. Less prone to errors
4. Faster reactivity since you don't need an additional system to talk changes.

Conclusion RedisOplog is a temporary solution until ChangeStreams comes to the mongodb node driver. The story behind it was noble, I've invested a lot of time in it, we managed to solve an important issue of Meteor, but the sad(or happy) reality is that once Notifications API comes to life, that is it, that is the way to go. It's the same thing they did with Blaze, it is a beautiful engine, it solves elegantly lots of problems, but React won, I am agnostic.

@raphaelarias @msavin @ramezrafla

At this stage, investing in marketing is futile. If someone needs to scale fast, I can offer Premium Support to them and make sure the system is bug free and working flawlessly, until NotificationsAPI gets implemented.

[ramezrafla]

Agreed, thanks @theodorDiaconu for taking leadership with Redis-oplog. We need to keep innovating and that's how we keep progressing. Some technology sticks, some doesn't, but the experience, reputation, and pedigree remain.

Disclaimer: haven't yet looked at changeStreams, taking at face value

[mpowaga]

According to this comment change streams has been implemented in 3.5.x release (just few days ago). I've been playing with 3.6.0-rc0 and it works really nice! I'm thinking of creating PoC replacement for oplog driver and comparing peformance of these two once I find some time.

[raphaelarias]

Awesome. We can soon start testing the new Change Streams API. Just as a reminder, for a wider release, there is still this issue to be closed: #152

[klaussner]

I've started to work on a Change Stream observe driver prototype last week but it turned out to be more challenging than the announcement blog post [1] suggested. :sweat_smile: There isn't much official documentation yet but the API specification [2], the talk "Using Change Streams to Keep Up with Your Data" [3], and the code of the Node.js driver [4] are good starting points if you want to learn more about Change Streams.

Here's what I found out:

Change Streams cannot be used to watch cursors. Think of them more as an API that is similar to the oplog but with some additional features, such as filtering changes using queries. Unfortunately, they don't provide the same real-time capabilities as RethinkDB or Firebase. I think that an implementation of Meteor's observe driver using Change Streams would be very similar to RedisOplog, with similar limitations.

Let's take a look at some examples that show why using Change Streams with Meteor is not that simple:

Query by _id

If documents are queried by _id (or multiple _ids with $in), Change Streams can be used very efficiently because they can filter out all changes to other documents. For example, the query

Documents.find('<_id>');

would initiate the following Change Stream in the observe driver:

collection.watch([{
  $match: { 'documentKey._id': '<_id>' }
}]);

By listening to all changes to the matched document, the server will only get notified if this document is updated or removed and can send the updates along to its connected clients.

Query by Arbitrary Fields

Unfortunately, querying by arbitrary fields (and perhaps using skip and limit) makes tracking changes a lot more complicated (that's what @theodorDiaconu describes as FLS queries in https://github.com/meteor/meteor-feature-requests/issues/158#issuecomment-335084684). For example, the query

Documents.find({ someField: '<some value>' });

requires that the observe driver maintains a set of matched document _ids. Otherwise, it couldn't notify its clients if a new document joins the set (as a result of an update) or leaves the set (as a result of a remove or an update). Just using one Change Stream as in the first example doesn't work here because the observe driver would miss important updates. For example, if a client wants to display a list of all tasks that aren't done yet, it would query todos documents like this:

Todos.find({ done: false });

The associated Change Stream would filter out documents whose done field isn't false:

Todos.watch([{
  $match: { 'fullDocument.done': false }
}]);

However, if the done field of one of the matched documents is changed to true, the server will never be notified of that change because the document doesn't match anymore! A trivial solution for this problem would be to just watch all changes, including the inverse of the query, but that wouldn't be an improvement over the current oplog tailing approach.

Change Stream Use Cases

The key difference between these two cases is the immutability of the queried field. Since _ids never change, documents cannot enter or leave the set of matched documents when they are updated. Matching on changing fields is not a use case that Change Streams were designed for (if the application wants to maintain state about the documents, which is the case for Meteor). [3, 18:45]

It's possible to match on unchanging fields other than _id (e.g., an authorId of a blog post document). However, it depends on the application if a field is immutable or not, so Meteor cannot assume immutablility for fields other than _id.

Performance

There are also some performance-related things to keep in mind:

• Using the database instead of the server to check if relevant fields of a document have changed can impact the performance of other database operations. The goal for MongoDB is to support 1000 simultaneous Change Streams per mongod process without impacting performance. [3, 7:40 and 29:13]
• Documents that are matched by more than one Change Stream are sent to the server multiple times, which could increase traffic between the database and the server if overlap between Change Streams isn't kept as small as possible (for example, by reusing them for identical queries).

References

[1] Blog: New Driver Features for MongoDB 3.6
[2] Change Streams API specification
[3] Video: Using Change Streams to Keep Up with Your Data
[4] 3.0.0 branch of node-mongodb-native

[theodorDiaconu]

The key difference between these two cases is the immutability of the queried field. Since _ids never change, documents cannot enter or leave the set of matched documents when they are updated. Matching on changing fields is not a use case that Change Streams were designed for. [3, 18:45]

This statement right here just brought the life back to RedisOplog. They never specified this in their docs. Oh well, time to announce the resurrection.

@msavin @ramezrafla @raphaelarias it seems that the mission hasn't ended yet.

[aogaili]

So basically the Change Stream API only return events pertaining to the subset of the document that satisfies the match condition and if a document field change that invalidate the match criteria (mutable query field) then no event will be generated. Is that correct?

Assuming the API returns an event when a document no longer meet the match condition, then would that solve this limitation? or are there any other major imitation that you've observed?

[mpowaga]

Change Streams are just watching oplog and return matching entries. Perhaps biggest performance gain is thanks to fullDocument option which can reduce mongo lookups.

Another limitation is that Change Streams can be used only when mongo server is running with --enableMajorityReadConcern option which requires WiredTiger storage.

[mitar]

The goal for MongoDB is to support 1000 simultaneous Change Streams per mongod process without impacting performance.

Hm, this is also a pretty low limit, wouldn't people agree?

Assuming the API returns an event when a document no longer meet the match condition

I think we should report this upstream.

Thank you @klaussner for looking into this!

[mitar]

Opened: https://jira.mongodb.org/browse/SERVER-31720

[mpowaga]

The goal for MongoDB is to support 1000 simultaneous Change Streams per mongod process without impacting performance.

Documents that are matched by more than one Change Stream are sent to the server multiple times, which could increase traffic between the database and the server if overlap between watchers isn't kept as small as possible (for example, by reusing watchers for identical queries).

Looks like in some cases oplog tailing would be more performant than Change Streams.

[msavin]

Maybe we need to forget about making all the things realtime like with oplog, and instead focus on creating a solution that aligns with MongoDB Change Streams and/or Redis?

For example, with Meteor today, subscribing to just one document in a collection is most efficient - and maybe we should move on from "massive" subscriptions.

[atcabral]

@mpowaga We are getting rid of the need for the startup option of --enableMajorityReadConcern in MongoDB 3.6 by making readConcern:Majority always available on every mongod.

[msavin]

To continue on the previous comment, my understanding is that:

• subscribing to just one document is way more efficient than subscribing to multiple documents
• getting multiple documents over a method is way faster than using a subscription.

If I were building a Meteor app today, I would probably use methods for big data sets and subscriptions to get one document at a time, for what I would call a reactive object. If we were to agree that this is the right approach, perhaps we should focus on creating a "LiveObject" package instead of replacing LiveQuery.

LiveObject can be powered by Redis (which supports persistent storage) or (I think) with MongoDB and ChangeStreams. It would essentially be one document stored in MongoDB or a JSON object stored in Redis. It can also used with Tracker to "trigger" a method call to reload static data when needed.

In the end, we can have:

• LiveQuery (via oplog)
• LiveObject (via redis or mongo)
• Static data (via methods)

As for scaling oplog, I'm not sure how reasonable it is. RethinkDB tried to make a real time database but it looks like it had limitations, and it looks like MDG abandoned it completely.

I think oplog tailing will always be great for small-medium custom apps, and maybe that's all it needs to be. Or perhaps, we should try to implement a scalable LiveObject solution before trying something far more complicated.

If there's interest, I can do my part by creating a page and spec for it.

Edit: Maybe a better name is LiveDocument, and it could be implemented on subscriptions that use findOne instead of find.

[s-devaney]

RethinkDB tried to make a real time database but it looks like it had limitations, and it looks like MDG abandoned it completely.

Perhaps it's time to re-open the discussion RE: RethinkDB?

[msavin]

@s-devaney I think that thing is on life support. Last I checked, it had a lot of issues, but I could be wrong. MongoDB and Redis look very healthy in comparison.

---

I just opened up a discussion focusing on the limited use case of Change Streams. It would be great to get everyone's opinion in there: https://forums.meteor.com/t/implementing-livedocument-as-an-alternative-to-livequery-discussion/40152/1

[msavin]

Interesting quote from MDG:

“Including id in queries can also help improve scaling when using oplog tailing. In general, running queries on id will be faster and have more predictable performance. If it is possible to rework your schema so that _ids are used more extensively, especially on hot collections, you may able to get orders of magnitude improvement in performance.”

Change Streams and Meteor are an amazing fit. Subscribing by _id is the most efficient way to use Pub/Sub, and it's exactly what Change Streams supports.

[klaussner]

@msavin In my experience, it's often impossible to restructure a schema (or app) so that it allows all documents to be queried by _id. To take advantage of database-level filtering with Change Streams, all queries (for a specific collection) in a container would have to use the _id field. As soon as you observe one non-_id query, the container must receive all changes* to all documents (no improvement compared to oplog tailing). Given that querying by _id is already quite fast with oplog tailing, I think that Change Streams wouldn't bring a significant performance or scalability improvement for the "query by _id" use case.

_{*It's possible to filter out irrelevant fields using an aggregation pipeline, so the amount of data sent per update could be reduced. However, there's a tradeoff to be made between less data sent per update and less duplication of updates.}

[msavin]

@klaussner for queries that cannot be fulfilled by _id only, Method's could and probably should be used instead.

With that said, Change Streams would bring several benefits here:

• Ability to focus on specific collections instead of whole database
• Ability to focus in on specific fields instead of all the fields
• Ability to notify server of events not related to pub/sub

All in all, it would reduce number of events coming in from database, which might be the most important thing. Additionally, it paves the way to make things even more efficient, i.e. if we were to create Meteor microservices for specific publications.

[raphaelarias]

@msavin I believe it would be better to always retrieve data form the server via Pub/Sub, instead of Methods. We do use methods for non-reactive things, but the official way it is still Pub/Sub and it's more organised that way, don't you think?

It's not always possible to query by _id, specially if new documents will be added. But in any case, we still have to handle non _id queries. Would a solution be: every time we do an Insert we validated if it would be part of a Pub's query and if yes, add it to the Pub? I tend to think that solution it's not scalable.

[msavin]

We can debate it for ages, most of it comes down to your engineering choices.

Speaking for myself, I would gladly adjust my approach to designing applications to leverage subscriptions by _id and Change Streams. It would give me better scale and performance at a lower cost - it's not bad when you get 3 out of 3.