search

10gen / mongo-orchestration

Apache License 2.0
7 stars 11 forks source link

Flaky auth failure on cluster setup #310

Open ShaneHarvey opened 10 months ago

ShaneHarvey commented 10 months ago

Flaky auth failure on cluster setup:

https://spruce.mongodb.com/task/mongo_python_driver_tests_python_version_rhel8_test_ssl__platform~rhel8_auth_ssl~auth_ssl_python_version~3.7_coverage~coverage_test_4.2_replica_set_8422edf3abced2b9492123bf72ac91ab3586a69d_23_12_07_21_46_46/logs?execution=0

 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37373, host=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.servers:342 - pid=37373, hostname=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37379, host=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.servers:342 - pid=37379, hostname=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,869 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37383, host=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,870 [DEBUG] mongo_orchestration.servers:342 - pid=37383, hostname=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,881 [ERROR] mongo_orchestration.apps:68 - <function rs_create at 0x7fef4a9a3920>
 [2023/12/07 14:07:00.236] Traceback (most recent call last):
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/__init__.py", line 66, in wrap
 [2023/12/07 14:07:00.236]     return f(*arg, **kwd)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 80, in rs_create
 [2023/12/07 14:07:00.236]     result = _rs_create(data)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 37, in _rs_create
 [2023/12/07 14:07:00.236]     rs_id = ReplicaSets().create(params)
 [2023/12/07 14:07:00.236]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 648, in create
 [2023/12/07 14:07:00.236]     repl = ReplicaSet(rs_params)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 120, in __init__
 [2023/12/07 14:07:00.236]     self.restart_with_auth()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 159, in restart_with_auth
 [2023/12/07 14:07:00.236]     self.restart(config_callback=add_auth)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 617, in restart
 [2023/12/07 14:07:00.236]     f.result()
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
 [2023/12/07 14:07:00.236]     return self.__get_result()
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
 [2023/12/07 14:07:00.236]     raise self._exception
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
 [2023/12/07 14:07:00.236]     result = self.fn(*self.args, **self.kwargs)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 449, in restart
 [2023/12/07 14:07:00.236]     return self.start(timeout)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 351, in start
 [2023/12/07 14:07:00.236]     self.run_command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 272, in run_command
 [2023/12/07 14:07:00.236]     result = getattr(self.connection.admin, mode)(command, name, **d)
 [2023/12/07 14:07:00.236]                      ^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 215, in connection
 [2023/12/07 14:07:00.236]     connected(c)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/common.py", line 130, in connected
 [2023/12/07 14:07:00.236]     client.admin.command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
 [2023/12/07 14:07:00.236]     return func(self, *args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/database.py", line 890, in command
 [2023/12/07 14:07:00.236]     with self.__client._conn_for_reads(read_preference, session) as (
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1330, in _conn_from_server
 [2023/12/07 14:07:00.236]     with self._checkout(server, session) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1252, in _checkout
 [2023/12/07 14:07:00.236]     with server.checkout(handler=err_handler) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1637, in checkout
 [2023/12/07 14:07:00.236]     conn = self._get_conn(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1756, in _get_conn
 [2023/12/07 14:07:00.236]     conn = self.connect(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1607, in connect
 [2023/12/07 14:07:00.236]     conn.authenticate()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1078, in authenticate
 [2023/12/07 14:07:00.236]     auth.authenticate(creds, self, reauthenticate=reauthenticate)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 625, in authenticate
 [2023/12/07 14:07:00.236]     auth_func(credentials, conn)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 530, in _authenticate_default
 [2023/12/07 14:07:00.236]     return _authenticate_scram(credentials, conn, "SCRAM-SHA-1")
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 257, in _authenticate_scram
 [2023/12/07 14:07:00.236]     res = conn.command(source, cmd)
 [2023/12/07 14:07:00.236]           ^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 322, in inner
 [2023/12/07 14:07:00.236]     return func(*args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 968, in command
 [2023/12/07 14:07:00.236]     return command(
 [2023/12/07 14:07:00.236]            ^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/network.py", line 192, in command
 [2023/12/07 14:07:00.236]     helpers._check_command_response(
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 230, in _check_command_response
 [2023/12/07 14:07:00.236]     raise OperationFailure(errmsg, code, response, max_wire_version)
 [2023/12/07 14:07:00.236] pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,886 [DEBUG] mongo_orchestration.apps:55 - send_result(500)
 [2023/12/07 14:07:00.240] curl: (22) The requested URL returned error: 500 Internal Server Error

Perhaps the user was rolled back in the restart process? We should double check that we are using write concern majority when adding users.

ShaneHarvey commented 10 months ago

We already create the user with the correct write concern:

        # Explicitly set write concern to number of data-bearing members.
        # If we add a user later, we need to guarantee that every node
        # has the user before we authenticate ('majority' is insufficient).
        self._write_concern = len(
            [m for m in members
             if not m.get('rsParams', {}).get('arbiterOnly')]
        )
ShaneHarvey commented 3 weeks ago

This error can happen when an existing cluster is running on port X and we try to start a new cluster on the same port.