search

rueckstiess / mtools

A collection of scripts to set up MongoDB test environments and parse and visualize MongoDB log files.
Apache License 2.0
1.88k stars 400 forks source link

mlaunch fails launching a sharded cluster on evergreen #691

Closed p-mongo closed 4 years ago

p-mongo commented 5 years ago

I ran this:

[2019/06/06 13:57:13.155] + mlaunch --dir /data/mci/f248d67c080a7afe47e162395af1375f/src/../drivers-tools/.evergreen/orchestration/db --setParameter enableTestCommands=1 --filePermissions 0666 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem --replicaset --sharded 2 --name ruby-driver-rs --auth --username alice --password wland
[2019/06/06 13:57:35.647] launching: "mongod" on port 27018
[2019/06/06 13:57:35.647] launching: "mongod" on port 27019
[2019/06/06 13:57:35.647] launching: "mongod" on port 27020
[2019/06/06 13:57:35.647] launching: "mongod" on port 27021
[2019/06/06 13:57:35.647] launching: "mongod" on port 27022
[2019/06/06 13:57:35.647] launching: "mongod" on port 27023
[2019/06/06 13:57:35.647] Traceback (most recent call last):
[2019/06/06 13:57:35.647] launching: config server on port 27024
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/src/../drivers-tools/.evergreen/orchestration/python/bin/mlaunch", line 10, in 
[2019/06/06 13:57:35.647] replica set 'configRepl' initialized.
[2019/06/06 13:57:35.647]     sys.exit(main())
[2019/06/06 13:57:35.647] replica set 'shard01' initialized.
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 2068, in main
[2019/06/06 13:57:35.647] replica set 'shard02' initialized.
[2019/06/06 13:57:35.647] launching: mongos on port 27017
[2019/06/06 13:57:35.647] adding shards. can take up to 30 seconds...
[2019/06/06 13:57:35.647]     tool.run()
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 542, in run
[2019/06/06 13:57:35.647]     getattr(self, self.args['command'])()
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 773, in init
[2019/06/06 13:57:35.647]     roles=roles)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/mtools/mlaunch/mlaunch.py", line 1708, in _add_user
[2019/06/06 13:57:35.647]     **opts)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 1041, in add_user
[2019/06/06 13:57:35.647]     session=session, **kwargs)
[2019/06/06 13:57:35.647]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 961, in _create_or_update_user
[2019/06/06 13:57:35.648]     self.command(command_name, name, session=session, **opts)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 532, in command
[2019/06/06 13:57:35.648]     codec_options, session=session, **kwargs)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/database.py", line 439, in _command
[2019/06/06 13:57:35.648]     client=self.__client)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/pool.py", line 522, in command
[2019/06/06 13:57:35.648]     self._raise_connection_failure(error)
[2019/06/06 13:57:35.648]   File "/data/mci/f248d67c080a7afe47e162395af1375f/drivers-tools/.evergreen/orchestration/python/pymongo/pool.py", line 677, in _raise_connection_failure
[2019/06/06 13:57:36.186]     raise error
pymongo.errors.NotMasterError: not master
[2019/06/06 13:57:36.186] Command failed: command encountered problem: exit status 1

Note that mlaunch said it may take up to 30 seconds to add shards, but then failed after 1 second due to a not master error.

Should server selection timeout be increased?

Environment

Software Version
mtools 1.5.3
MongoDB server 4.1.9
Operating system ubuntu1404

[2019/06/06 13:57:13.052] Successfully installed mtools-1.5.3 psutil-5.4.2 pymongo-3.6.1 python-dateutil-2.7.0 six-1.12.0

p-mongo commented 4 years ago

Any update/thoughts on this? I am moving all of our configurations to mlaunch and this issue is a significant problem because it fails the entire test run.

[2020/02/14 00:44:51.147] + mlaunch --dir /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023
[2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024
[2020/02/14 00:45:15.439] launching: config server on port 27025
[2020/02/14 00:45:15.439] replica set 'configRepl' initialized.
[2020/02/14 00:45:15.439] replica set 'shard01' initialized.
[2020/02/14 00:45:15.439] replica set 'shard02' initialized.
[2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017
[2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018
[2020/02/14 00:45:15.439] adding shards. can take up to 30 seconds...
[2020/02/14 00:45:15.935] Traceback (most recent call last):
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 8, in 
[2020/02/14 00:45:15.935]     sys.exit(main())
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 2068, in main
[2020/02/14 00:45:15.935]     tool.run()
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 542, in run
[2020/02/14 00:45:15.935]     getattr(self, self.args['command'])()
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 773, in init
[2020/02/14 00:45:15.935]     roles=roles)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 1708, in _add_user
[2020/02/14 00:45:15.935]     **opts)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1339, in add_user
[2020/02/14 00:45:15.935]     session=session, **kwargs)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1253, in _create_or_update_user
[2020/02/14 00:45:15.935]     self.command(command_name, name, session=session, **opts)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 733, in command
[2020/02/14 00:45:15.935]     codec_options, session=session, **kwargs)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 630, in _command
[2020/02/14 00:45:15.935]     client=self.__client)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 618, in command
[2020/02/14 00:45:15.935]     self._raise_connection_failure(error)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
[2020/02/14 00:45:15.935]     user_fields=user_fields)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
[2020/02/14 00:45:15.935]     parse_write_concern_error=parse_write_concern_error)
[2020/02/14 00:45:15.935]   File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/helpers.py", line 136, in _check_command_response
[2020/02/14 00:45:15.935]     raise NotMasterError(errmsg, response)
[2020/02/14 00:45:15.935] pymongo.errors.NotMasterError: not master
[2020/02/14 00:45:15.950] Command failed: command encountered problem: error waiting on process '1b99b8da-e02f-4b16-9f75-f37f58a0df05': exit status 1

The diagnostics here could be better. Is mlaunch requesting a primary server? If yes, the error should happen after server selection timeout. Does mlaunch change server selection timeout from the default? I assume the default value is more than 1 second or so. I am not seeing primary changes in server logs - each replica set elected its primary one time.

What other information can I provide here?

Run log: http://speed:9393/repos/mongodb/mongo-ruby-driver/pulls/1691/evergreen-log/mongo_ruby_driver_jruby_auth__mongodb_version~4.2_topology~sharded_cluster_auth_and_ssl~auth_and_ssl_ruby~jruby_9.2_os~ubuntu1604_patch_feadf5ebbba8774f6deaba0dc9cb05fc18d2caa7_5e463396e3c3311d0f6cd534_20_02_14_05_43_51#first-failure

Server logs: http://speed:9393/eg/mongo-ruby-driver/versions/5e4634db7742ae0eebc9db86/builds/mongo_ruby_driver_jruby_auth__mongodb_version~4.2_topology~sharded_cluster_auth_and_ssl~auth_and_ssl_ruby~jruby_9.2_os~ubuntu1604_patch_feadf5ebbba8774f6deaba0dc9cb05fc18d2caa7_5e463396e3c3311d0f6cd534_20_02_14_05_43_51/artifact-logs

p-mongo commented 4 years ago

Repro: https://github.com/p-mongo/tests/tree/master/mlaunch-not-master

p-mongo commented 4 years ago

Actually this issue is still happening, my fix either didn't do anything or was not sufficient.

ShaneHarvey commented 4 years ago

I believe the bug is that mlaunch connects directly to a single mongod to add the initial user here: https://github.com/p-mongo/mtools/blob/80be43b/mtools/mlaunch/mlaunch.py#L1655

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port)

It should be connecting to the replica set as a whole, like this:

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port, replicaSet=shard_replica_set_name)

or:

    def _add_user(self, port, name, password, database, roles):
        con = self.client('localhost:%i'%port)
        ismaster = con['admin'].command('isMaster')
        set_name = ismaster.get('setName')
        if set_name:
            con = self.client('localhost:%i'%port, replicaSet=set_name)
p-mongo commented 4 years ago

I applied @ShaneHarvey 's fix in https://github.com/p-mongo/mtools/commits/wait-for-rs, but now I am frequently getting this error:

[2020/02/24 18:21:13.011] + eval mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/24 18:21:13.011] ++ mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem
[2020/02/24 18:21:34.357] Finished processing dependencies for mtools-legacy==1.5.4
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023
[2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024
[2020/02/24 18:21:34.357] launching: config server on port 27025
[2020/02/24 18:21:34.357] replica set 'configRepl' initialized.
[2020/02/24 18:21:34.357] replica set 'shard01' initialized.
[2020/02/24 18:21:34.357] replica set 'shard02' initialized.
[2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017
[2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018
[2020/02/24 18:21:34.357] adding shards. can take up to 30 seconds...
[2020/02/24 18:21:34.357] Traceback (most recent call last):
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 9, in 
[2020/02/24 18:21:34.357]     load_entry_point('mtools-legacy==1.5.4', 'console_scripts', 'mlaunch')()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 2073, in main
[2020/02/24 18:21:34.357]     tool.run()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 542, in run
[2020/02/24 18:21:34.357]     getattr(self, self.args['command'])()
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 773, in init
[2020/02/24 18:21:34.357]     roles=roles)
[2020/02/24 18:21:34.357]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 1701, in _add_user
[2020/02/24 18:21:34.358]     v = con['admin'].command('isMaster').get('maxWireVersion', 0)
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/database.py", line 730, in command
[2020/02/24 18:21:34.358]     read_preference, session) as (sock_info, slave_ok):
[2020/02/24 18:21:34.358]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
[2020/02/24 18:21:34.358]     return self.gen.next()
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
[2020/02/24 18:21:34.358]     server = self._select_server(read_preference, session)
[2020/02/24 18:21:34.358]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
[2020/02/24 18:21:34.359]     server = topology.select_server(server_selector)
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 235, in select_server
[2020/02/24 18:21:34.359]     address))
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 193, in select_servers
[2020/02/24 18:21:34.359]     selector, server_timeout, address)
[2020/02/24 18:21:34.359]   File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
[2020/02/24 18:21:34.359]     self._error_message(selector))
[2020/02/24 18:21:34.359] pymongo.errors.ServerSelectionTimeoutError: No replica set members match selector "Primary()"
[2020/02/24 18:21:34.869] Command failed: command encountered problem: error waiting on process '95343b3a-3e70-403f-884f-a811b90cecc2': exit status 1

I tried setting the server selection timeout explicitly in https://github.com/p-mongo/mtools/commit/b932e763fb59b9b20972036ea792e0d58b024aba since the error message does not indicate what the timeout is, and you can see in the above log that there is a 20 second gap between the second and the third line indicating the client did wait for some time (timestamps are inaccurately reported by evergreen, if you are wondering why a bunch of log entries have the same timestamp). The timeout seems to have not had any effect.

ShaneHarvey commented 4 years ago

Can you try adding serverSelectionTimeoutMS=10000 to the replica set client too?: https://github.com/p-mongo/mtools/blob/b932e763fb59b9b20972036ea792e0d58b024aba/mtools/mlaunch/mlaunch.py#L1700