Allow write concern option to be specified for database operations

[mitar]

Migrated from: meteor/meteor#1547

[msavin]

I think this is very important topic. Most developers do not know that by default, MongoDB operates in a fire-and-forget way. I've been hosting my app on Compose, which shares resources with other databases and does not use write concerns. I've noticed it missing database operations on an app that has minimal writes and maybe 5 cu's at most.

As a result, I'm going to be using MongoDB Atlas from now on, because they let you set Write Concerns.

My understanding is that setting Write Concerns modifies how the database operates. Obviously, it'll be able to handle less writes per second but with a better guarantee of those writes succeeding.

I wonder if Meteor can or does provide some kind of verification of writes, or some kind of response from MongoDB? My understanding is that currently, Mongo.Collection gives back an _id, but that _id is just a randomly generated string, and does not guarantee that the write was successful, or that there was no write conflict as a result of a document with that _id.

An app is only as good as the integrity of the data, and it would be great if Meteor can crank it up from 90% to 100% in this department.

[hwillson]

Hi @msavin - a few corrections:

Most developers do not know that by default, MongoDB operates in a fire-and-forget way.

This isn’t quite accurate. Meteor uses the Mongo default write concern, which is w: 1 - acknowledge that the write operation has succeeded.

I've been hosting my app on Compose, which shares resources with other databases and does not use write concerns.

Write concerns are set when you perform your document insert, update, etc. Compose.io definitely allows/uses write concerns.

My understanding is that setting Write Concerns modifies how the database operates. Obviously, it'll be able to handle less writes per second but with a better guarantee of those writes succeeding.

Sort of - you’re not changing how the database operates; you’re just changing your result handling expectation. If you want better write performance, then you would use a write concern of w: 0 - it’s faster because you’re telling Mongo you don’t care about waiting around to see if the write succeeded, so you can move on to your next operation faster.

I wonder if Meteor can or does provide some kind of verification of writes, or some kind of response from MongoDB?

It sure can - as a matter of fact, it’s doing it already. Since Meteor is using a default write concern of w: 1, if the write doesn’t succeed, then you’ll know about it. Either an exception will be thrown, or if you’re passing a callback into the collection call, it will receive an error object.

The above all being said, this issue isn’t about Meteor + Mongo’s data integrity (your data should be fine) - it’s about allowing people to (easily) adjust the Mongo query write concern that’s used. So providing a way to override the default.

Just to add - Mongo has some data integrity issues of its own that write concerns cannot fix (although they're making progress), but those issues are outside the control of Meteor.

Hope this helps!

[alimgafar]

The link to the Jespen report is relevant prior to MongoDB 3.4, and yet another reason I think Meteor should be released with the current blessed version of MongoDB. See https://jepsen.io/analyses/mongodb-3-4-0-rc3

Full disclosure: I'm a former MongoDB employee.

[mitar]

Ohh, @alimgafar, this was amazing read. I think this might even influence Meteor oplog tailing. I think it still uses v0 oplog for its tailing. Maybe Meteor should also switch to v1 and require a database with those fixes from Jepsen. @glasser, what do you think?

[glasser]

It's been a couple years since I've personally actively worked on the Meteor framework, but yes, this all does sound reasonable (as I've thought all along, see my "I agree" in the previous thread). I think Meteor's approach to API design is more pragmatic these days and patches are more likely to be accepted quickly.

[mitar]

@hwillson, @glasser: I think Meteor should ship with majority default write concern, and make sure no dirty reads end up the oplog chain and trigger any observes which are then invalidated. I think default Meteor should come with full correctness, even if it is at expanse of performance, and then users can use lesser concern if they need high throughput writes but do not care if they loose some writes.

[mitar]

I think the hard change might be assuring that the oplog does not end up processing stale reads, changes which later on get invalidated because of the primary switch. Maybe the solution is that oplog always connects to a secondary and processes only committed changes, but not sure if that is really doable. How you know from just observing the oplog what got committed by the majority?

Maybe MDG could hire Jespen to evaluate Meteor's oplog handling?

[mitar]

(Or that should be a different issue: feature request for read concern option.)

[glasser]

At least in theory, the oplog observe driver re-polls all observes when it observes a failover. This might not be perfectly implemented.

I would agree that majority is a good default write concern, but as I mentioned, my hands are full with Galaxy these days.