Upgrade to MongoDB 4.2 support

[benlavalley]

Now that MongoDB 4.2 is released, Meteor should be updated to support it.

It introduces a few new key features, namely faster replica set failover handling and initial sync, lower cpu / higher compression algorithm "Zstandard" / Zstd, wildcard support for indexes (huge), and online index builds (also huge).

The NodeJs driver for Mongodb 4.2 was released 3 days ago, so hopefully we can see it included in Meteor soon.

@benjamn <3 :)

[mitar]

Is there anything necessary on Meteor side, except for the driver update?

[benlavalley]

Hopefully not. I looked through and see a lot of refactoring, rewrites and some new features, but didn't spot any breaking API changes (though there is a new mechanism for topology selection introduced in 3.2, that will be mandatory at some point in the future in 4.x). I was debating trying it out locally if I can, if only for the wildcard index support... I was just about to re-factor indexes on a massive collection but will hold off until I can use 4.2.

Going to try the trick to edit the npm-mongo wrapper package locally with a development database and see how it goes.

[benlavalley]

Few days now using the new mongo 3.3.0 driver with 4.0 and 4.2 mongo DBs - so far so good. I have to hold off on updating my Mongo featureCompatibilityVersion to 4.2 until Mongo's OpsManager backup support catches up with 4.2.

For anyone curious, it was just a matter of cloning the Meteor npm-mongo wrapper package into /packages, and updating package.js for the mongodb 3.3.0. You can verify the driver update with the MongoInternals.NpmModules.mongodb.version command.

[StorytellerCZ]

@benlavalley Can you submit a PR with your update. It is a quick improvement, but the first step to add additional features that have been requested by other feature requests.

[benlavalley]

@StorytellerCZ Will update my production instances to feature compatibility of 4.2 (one which Meteor monitors the oplog for, another without oplog monitoring enabled) and provided all goes well I'll get it submitted.

I also browsed the Meteor source tree to see if I could figure out getting Meteor's default development database updated to 4.2 as well.

Lots of reasons to look at MongoDB 4.2 especially for production apps.

[StorytellerCZ]

I took a look at the code quickly and I don't see anything that would create an issue (but testing never hurts).

[klaussner]

Updating the bundled MongoDB binaries (mongod and mongo) is going to be more difficult this time because MongoDB stopped releasing generic Linux builds, which Meteor currently uses. The advantage of these builds was that they run on any Linux distribution. Starting with MongoDB 4.2, you can only download builds that work on specific versions of Linux distributions. For example, the build for Ubuntu 16.04 won't work on Ubuntu 18.04 and vice versa because the operating systems ship with different OpenSSL versions.

I've downloaded some of the MongoDB builds and compared their dependencies:

Last available generic build (MongoDB 4.0.12):

libresolv.so.2
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

Debian 9, Ubuntu 18.04 (MongoDB 4.2.0):

+ libcurl.so.4
libresolv.so.2
+ libcrypto.so.1.1
+ libssl.so.1.1
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

Ubuntu 16.04, SUSE 12 (MongoDB 4.2.0):

+ libcurl.so.4
libresolv.so.2
+ libcrypto.so.1.0.0
+ libssl.so.1.0.0
libdl.so.2
librt.so.1
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

As you can see in the diffs, the builds link to different versions of libcrypto and libssl (both are part of OpenSSL). I think we have three options to solve this problem:

1. Stop bundling MongoDB with Meteor and let developers install the required version manually.
2. Bundle a MongoDB build that links to the most recent versions of its dependencies and make it the responsibility of developers to install the necessary libraries (even if their operating systems don't provide them in their official repositories).
3. Figure out a way to build MongoDB from source without SSL support, which would result in binaries that are similar to the generic builds that Meteor currently uses.

I think (1) and (2) would impact developer experience too much and (3) would require a fair amount of additional work. Any suggestions? 😅

[mitar]

Why not simply bundle dependencies as well and when starting mongodb process during development, we configure things so that it looks in local directory for those dependencies instead of system directory, by setting LD_LIBRARY_PATH.

[klaussner]

That could work for the OpenSSL libraries but I'm not sure about libcurl because it depends on other dynamic libraries that we would probably also have to include (https://packages.debian.org/de/sid/libcurl4).

[mitar]

Sure. You might have to do it recursively. But there is a finite amount of them. :-)

[klaussner]

I tried to build MongoDB from source and it was easier than expected. This Docker image contains all build dependencies and scripts: https://github.com/klaussner/mongodb-builder. The mongod and mongo binaries for MongoDB 4.2.0 can be found in the release tab.

[benjamn]

@klaussner Thanks so much for doing this investigation! I think building from source is a reasonable path, especially since this is just the development server (so SSL doesn't matter as much), and assuming the Mongo build is decoupled from the dev bundle build, so we don't have to rebuild Mongo every time we rebuild the dev bundle. These new Linux-distribution-specific builds/URLs really are a pain!

[afrokick]

Should be closed @filipenevola ?