test_charm.py::test_leader_is_primary_on_deployment failed in SolQA env

Steps to reproduce

Deploy 3 instances of MongoDB on top of MAAS cloud, with juju 3.3.4, with tls enabled.
run MongoDB integration tests, for tests/integration/test_charm.py only. Other test would pass, except test_leader_is_primary_on_deployment failed instead.

Expected behavior

Actual behavior

Versions

Operating system: Ubuntu 22.04

Juju CLI: 3.4.2

Juju agent: 3.4.2

Charm revision: 6/stable rev 164

LXD: N/A

Log output

Juju debug log: unit-mongodb-0: 18:27:51 ERROR unit.mongodb/0.juju-log _on_secret_remove: Secret secret:cofb3kvugmhkof8mmfmg seems to have no observers, could be removed
unit-mongodb-0: 18:27:54 ERROR unit.mongodb/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-0/charm/./src/charm.py", line 638, in set_password
mongo.set_user_password(username, password)
File "/var/lib/juju/agents/unit-mongodb-0/charm/lib/charms/mongodb/v0/mongodb.py", line 312, in set_user_password
self.client.admin.command(
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/_csot.py", line 105, in csot_wrapper
return func(self, *args, **kwargs)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/database.py", line 809, in command
return self._command(
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/database.py", line 688, in _command
return sock_info.command(
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/pool.py", line 767, in command
return command(
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/network.py", line 166, in command
helpers._check_command_response(
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/pymongo/helpers.py", line 181, in _check_command_response
raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Error preflighting UTF-8 conversion: U_STRING_NOT_TERMINATED_WARNING, full error: {'ok': 0.0, 'errmsg': 'Error preflighting UTF-8 conversion: U_STRING_NOT_TERMINATED_WARNING', 'code': 50687, 'codeName': 'Location50687', '$clusterTime': {'clusterTime ': Timestamp(1713292074, 2), 'signature': {'hash': b'\x00\x91\xe4txs\x02p\x03\xa6\\x8e\x0f:W\x02\x8ca\xd4\xd9', 'keyId': 7358514391030956038}}, 'operationTime': Timestamp(1713292074, 2)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-0/charm/./src/charm.py", line 607, in _on_set_password
secret_id = self.set_password(username, new_password)
File "/var/lib/juju/agents/unit-mongodb-0/charm/./src/charm.py", line 644, in set_password
raise SetPasswordError(f"Failed changing the password: {e}")
SetPasswordError: Failed changing the password: Error preflighting UTF-8 conversion: U_STRING_NOT_TERMINATED_WARNING, full error: {'ok': 0.0, 'errmsg': 'Error preflighting UTF-8 conversion: U_STRING_NOT_TERMINATED_WARNING', 'code': 50687, 'codeName': 'Location50687', '$clusterTime' : {'clusterTime': Timestamp(1713292074, 2), 'signature': {'hash': b'\x00\x91\xe4txs\x02p\x03\xa6\\x8e\x0f:W\x02\x8ca\xd4\xd9', 'keyId': 7358514391030956038}}, 'operationTime': Timestamp(1713292074, 2)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mongodb-0/charm/./src/charm.py", line 1526, in
main(MongodbOperatorCharm)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/main.py", line 441, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
event_to_emit.emit(args, kwargs)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/framework.py", line 344, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/framework.py", line 833, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/framework.py", line 922, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-mongodb-0/charm/./src/charm.py", line 609, in _on_set_password
event.fail(e)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/charm.py", line 185, in fail
self.framework.model._backend.action_fail(message)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/model.py", line 2942, in action_fail
self._run('action-fail', message)
File "/var/lib/juju/agents/unit-mongodb-0/charm/venv/ops/model.py", line 2693, in _run
result = subprocess.run(args, kwargs) # type: ignore
File "/usr/lib/python3.10/subprocess.py", line 503, in run
with Popen(popenargs, **kwargs) as process:
File "/usr/lib/python3.10/subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1796, in _execute_child
self.pid = _posixsubprocess.fork_exec(
TypeError: expected str, bytes or os.PathLike object, not SetPasswordError

INFO pytest_operator.plugin:plugin.py:904 Forgetting main...

============================================================================= FAILURES ============================================================================== [65/1869] _ test_leader_is_primary_on_deployment __
Traceback (most recent call last):
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 340, in from_call
result: Optional[TResult] = func()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 240, in
lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 138, in _multicall
raise exception.with_traceback(exception.traceback)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
yield from thread_exception_runtest_hook()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
yield from unraisable_exception_runtest_hook()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 849, in pytest_runtest_call
yield from self._runtest_for(item, "call")
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 832, in _runtest_for
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 883, in pytest_runtest_call
return (yield)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/skipping.py", line 256, in pytest_runtest_call
return (yield)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(args)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 182, in pytest_runtest_call
raise e
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 172, in pytest_runtest_call
item.runtest()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1772, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 181, in _multicall
return outcome.get_result()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 99, in get_result
raise exc.with_traceback(exc.traceback)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(args)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 195, in pytest_pyfunc_call
result = testfunction(testargs)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 532, in inner
_loop.run_until_complete(task)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
============================================================================= FAILURES ============================================================================== [65/1869] _ test_leader_is_primary_on_deployment __
Traceback (most recent call last):
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 340, in from_call
result: Optional[TResult] = func()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 240, in
lambda: runtest_hook(item=item, kwds), when=when, reraise=reraise
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 138, in _multicall
raise exception.with_traceback(exception.traceback)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
yield from thread_exception_runtest_hook()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
yield from unraisable_exception_runtest_hook()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 849, in pytest_runtest_call
yield from self._runtest_for(item, "call")
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/logging.py", line 832, in _runtest_for
yield
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/capture.py", line 883, in pytest_runtest_call
return (yield)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall
teardown.throw(exception) # type: ignore[union-attr]
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/skipping.py", line 256, in pytest_runtest_call
return (yield)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(args)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 182, in pytest_runtest_call
raise e
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/runner.py", line 172, in pytest_runtest_call
item.runtest()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 1772, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 181, in _multicall
return outcome.get_result()
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_result.py", line 99, in get_result
raise exc.with_traceback(exc.traceback)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(args)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/_pytest/python.py", line 195, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/pytest_asyncio/plugin.py", line 532, in inner
_loop.run_until_complete(task)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
====================================================================== short test summary info ====================================================================== FAILED tests/integration/test_charm.py::test_leader_is_primary_on_deployment - AssertionError: Leader is not primary ======================================================== 1 failed, 14 passed, 1 skipped in 144.09s (0:02:24) ======================================================== Task was destroyed but it is pending! task: <Task pending name='Task-50' coro=<WebSocketCommonProtocol.keepalive_ping() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:1254> wait_for=> Fatal error on SSL transport protocol: <asyncio.sslproto.SSLProtocol object at 0x7f1b44729780> transport: <_SelectorSocketTransport closing fd=10> Traceback (most recent call last): File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write n = self._sock.send(data) OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog self._transport.write(chunk) File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write self._fatal_error(exc, 'Fatal write error on socket transport') File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error self._force_close(exc) File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close self._loop.call_soon(self._call_connection_lost, exc) File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon self._check_closed() File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Task was destroyed but it is pending! task: <Task pending name='Task-49' coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:963> wait_for= cb=[Task.task_wakeup(), _wait.._on_completion() at /usr/lib/python3.10/asyncio/tasks.py:475]> Task was destroyed but it is pending! task: <Task pending name='Task-51' coro=<WebSocketCommonProtocol.close_connection() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:1301> wait_for=<Task pending name='Task-49' coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:963> wait_for= cb=[Task.task_wakeup(), _wait.._on_completion() at /usr/lib/python3.10/asyncio/tasks.py:475]>> Task was destroyed but it is pending! task: <Task pending name='Task-57' coro=<Connection._pinger() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py:589> wait_for=> Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x7f1b44697530> Traceback (most recent call last): File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1337, in close_connection await self.close_transport() File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1355, in close_transport if await self.wait_for_connection_lost(): File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py", line 1379, in wait_for_connection_lost async with asyncio_timeout(self.close_timeout): File "/home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/async_timeout.py", line 74, in timeout loop = asyncio.get_running_loop() RuntimeError: no running event loop Task was destroyed but it is pending! task: <Task pending name='Task-52' coro=<Connection._receiver() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/juju/client/connection.py:541> wait_for=> Task was destroyed but it is pending! task: <Task pending name='Task-1441' coro=<WebSocketCommonProtocol.recv() running at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/websockets/legacy/protocol.py:552> wait_for= cb=[create_task_with_handler.._task_result_exp_handler(task_name='tmp', logger=<Logger juju....ion (WARNING)>)() at /home/ubuntu/mongodb-test-repo/.tox/integration/lib/python3.10/site-packages/juju/jasyncio.py:50]> Task was destroyed but it is pending! task: <Task pending name='Task-1442' coro=<Event.wait() running at /usr/lib/python3.10/asyncio/locks.py:214> wait_for=> integration: exit 1 (144.79 seconds) /home/ubuntu/mongodb-test-repo> pytest -v --tb native --log-cli-level=INFO -s --ignore=/home/ubuntu/mongodb-test-repo/tests/unit/ -v --no-deploy --controller foundations-maas --model mongodb -m 'not unstable' tests/integration/test_charm.py pid=35271 integration: FAIL code 1 (144.83=setup[0.04]+cmd[144.79] seconds) evaluation failed :( (144.89 seconds)

canonical / mongodb-operator