Closed p-mongo closed 4 years ago
Any update/thoughts on this? I am moving all of our configurations to mlaunch and this issue is a significant problem because it fails the entire test run.
[2020/02/14 00:44:51.147] + mlaunch --dir /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019 [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020 [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021 [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022 [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023 [2020/02/14 00:45:15.439] launching: "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024 [2020/02/14 00:45:15.439] launching: config server on port 27025 [2020/02/14 00:45:15.439] replica set 'configRepl' initialized. [2020/02/14 00:45:15.439] replica set 'shard01' initialized. [2020/02/14 00:45:15.439] replica set 'shard02' initialized. [2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017 [2020/02/14 00:45:15.439] launching: /data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018 [2020/02/14 00:45:15.439] adding shards. can take up to 30 seconds... [2020/02/14 00:45:15.935] Traceback (most recent call last): [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 8, in[2020/02/14 00:45:15.935] sys.exit(main()) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 2068, in main [2020/02/14 00:45:15.935] tool.run() [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 542, in run [2020/02/14 00:45:15.935] getattr(self, self.args['command'])() [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 773, in init [2020/02/14 00:45:15.935] roles=roles) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/mtools/mlaunch/mlaunch.py", line 1708, in _add_user [2020/02/14 00:45:15.935] **opts) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1339, in add_user [2020/02/14 00:45:15.935] session=session, **kwargs) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 1253, in _create_or_update_user [2020/02/14 00:45:15.935] self.command(command_name, name, session=session, **opts) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 733, in command [2020/02/14 00:45:15.935] codec_options, session=session, **kwargs) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/database.py", line 630, in _command [2020/02/14 00:45:15.935] client=self.__client) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 618, in command [2020/02/14 00:45:15.935] self._raise_connection_failure(error) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command [2020/02/14 00:45:15.935] user_fields=user_fields) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/network.py", line 167, in command [2020/02/14 00:45:15.935] parse_write_concern_error=parse_write_concern_error) [2020/02/14 00:45:15.935] File "/data/mci/b29a65cc6eb09ac04181e7c52cac4ecd/drivers-tools/.evergreen/orchestration/venv/lib/python3.7/site-packages/pymongo/helpers.py", line 136, in _check_command_response [2020/02/14 00:45:15.935] raise NotMasterError(errmsg, response) [2020/02/14 00:45:15.935] pymongo.errors.NotMasterError: not master [2020/02/14 00:45:15.950] Command failed: command encountered problem: error waiting on process '1b99b8da-e02f-4b16-9f75-f37f58a0df05': exit status 1
The diagnostics here could be better. Is mlaunch requesting a primary server? If yes, the error should happen after server selection timeout. Does mlaunch change server selection timeout from the default? I assume the default value is more than 1 second or so. I am not seeing primary changes in server logs - each replica set elected its primary one time.
What other information can I provide here?
Actually this issue is still happening, my fix either didn't do anything or was not sufficient.
I believe the bug is that mlaunch connects directly to a single mongod to add the initial user here: https://github.com/p-mongo/mtools/blob/80be43b/mtools/mlaunch/mlaunch.py#L1655
def _add_user(self, port, name, password, database, roles):
con = self.client('localhost:%i'%port)
It should be connecting to the replica set as a whole, like this:
def _add_user(self, port, name, password, database, roles):
con = self.client('localhost:%i'%port, replicaSet=shard_replica_set_name)
or:
def _add_user(self, port, name, password, database, roles):
con = self.client('localhost:%i'%port)
ismaster = con['admin'].command('isMaster')
set_name = ismaster.get('setName')
if set_name:
con = self.client('localhost:%i'%port, replicaSet=set_name)
I applied @ShaneHarvey 's fix in https://github.com/p-mongo/mtools/commits/wait-for-rs, but now I am frequently getting this error:
[2020/02/24 18:21:13.011] + eval mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem [2020/02/24 18:21:13.011] ++ mlaunch --dir /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/db --binarypath /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin --setParameter enableTestCommands=1 --setParameter diagnosticDataCollectionEnabled=false --replicaset --sharded 2 --name ruby-driver-rs --mongos 2 --auth --username bob --password pwd123 --sslMode requireSSL --sslPEMKeyFile spec/support/certificates/server-second-level-bundle.pem --sslCAFile spec/support/certificates/ca.crt --sslClientCertificate spec/support/certificates/client.pem [2020/02/24 18:21:34.357] Finished processing dependencies for mtools-legacy==1.5.4 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27019 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27020 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27021 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27022 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27023 [2020/02/24 18:21:34.357] launching: "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongod" on port 27024 [2020/02/24 18:21:34.357] launching: config server on port 27025 [2020/02/24 18:21:34.357] replica set 'configRepl' initialized. [2020/02/24 18:21:34.357] replica set 'shard01' initialized. [2020/02/24 18:21:34.357] replica set 'shard02' initialized. [2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27017 [2020/02/24 18:21:34.357] launching: /data/mci/b815ceb4d907599e99cd6c1e46dcbf67/src/../drivers-tools/.evergreen/orchestration/mdb/mongodb-linux-x86_64-enterprise-ubuntu1604-4.2.3/bin/mongos on port 27018 [2020/02/24 18:21:34.357] adding shards. can take up to 30 seconds... [2020/02/24 18:21:34.357] Traceback (most recent call last): [2020/02/24 18:21:34.357] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/bin/mlaunch", line 9, in[2020/02/24 18:21:34.357] load_entry_point('mtools-legacy==1.5.4', 'console_scripts', 'mlaunch')() [2020/02/24 18:21:34.357] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 2073, in main [2020/02/24 18:21:34.357] tool.run() [2020/02/24 18:21:34.357] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 542, in run [2020/02/24 18:21:34.357] getattr(self, self.args['command'])() [2020/02/24 18:21:34.357] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 773, in init [2020/02/24 18:21:34.357] roles=roles) [2020/02/24 18:21:34.357] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/mtools_legacy-1.5.4-py2.7.egg/mtools/mlaunch/mlaunch.py", line 1701, in _add_user [2020/02/24 18:21:34.358] v = con['admin'].command('isMaster').get('maxWireVersion', 0) [2020/02/24 18:21:34.358] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/database.py", line 730, in command [2020/02/24 18:21:34.358] read_preference, session) as (sock_info, slave_ok): [2020/02/24 18:21:34.358] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ [2020/02/24 18:21:34.358] return self.gen.next() [2020/02/24 18:21:34.358] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads [2020/02/24 18:21:34.358] server = self._select_server(read_preference, session) [2020/02/24 18:21:34.358] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1253, in _select_server [2020/02/24 18:21:34.359] server = topology.select_server(server_selector) [2020/02/24 18:21:34.359] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 235, in select_server [2020/02/24 18:21:34.359] address)) [2020/02/24 18:21:34.359] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 193, in select_servers [2020/02/24 18:21:34.359] selector, server_timeout, address) [2020/02/24 18:21:34.359] File "/data/mci/b815ceb4d907599e99cd6c1e46dcbf67/drivers-tools/.evergreen/orchestration/venv/local/lib/python2.7/site-packages/pymongo/topology.py", line 209, in _select_servers_loop [2020/02/24 18:21:34.359] self._error_message(selector)) [2020/02/24 18:21:34.359] pymongo.errors.ServerSelectionTimeoutError: No replica set members match selector "Primary()" [2020/02/24 18:21:34.869] Command failed: command encountered problem: error waiting on process '95343b3a-3e70-403f-884f-a811b90cecc2': exit status 1
I tried setting the server selection timeout explicitly in https://github.com/p-mongo/mtools/commit/b932e763fb59b9b20972036ea792e0d58b024aba since the error message does not indicate what the timeout is, and you can see in the above log that there is a 20 second gap between the second and the third line indicating the client did wait for some time (timestamps are inaccurately reported by evergreen, if you are wondering why a bunch of log entries have the same timestamp). The timeout seems to have not had any effect.
Can you try adding serverSelectionTimeoutMS=10000
to the replica set client too?: https://github.com/p-mongo/mtools/blob/b932e763fb59b9b20972036ea792e0d58b024aba/mtools/mlaunch/mlaunch.py#L1700
I ran this:
Note that mlaunch said it may take up to 30 seconds to add shards, but then failed after 1 second due to a not master error.
Should server selection timeout be increased?
Environment
[2019/06/06 13:57:13.052] Successfully installed mtools-1.5.3 psutil-5.4.2 pymongo-3.6.1 python-dateutil-2.7.0 six-1.12.0