search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
800 stars 54 forks source link

sqlite3.OperationalError: database is locked #203

Closed christianbundy closed 7 months ago

christianbundy commented 1 year ago

Hi there! I'm looking into using pytest-testmon, but have bumped into this error a few times:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/_pytest/main.py", line 266, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1037, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/pytest_testmon.py", line 189, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/pytest_testmon.py", line 126, in init_testmon_data
INTERNALERROR>     testmon_data = TestmonData(
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/testmon_core.py", line 139, in __init__
INTERNALERROR>     self.db = db.DB(self.datafile, self.environment)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/db.py", line 49, in __init__
INTERNALERROR>     connection = connect(datafile)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/db.py", line 38, in connect
INTERNALERROR>     connection.execute("PRAGMA synchronous = OFF")
INTERNALERROR> sqlite3.OperationalError: database is locked

I'm running watchdog pytest.main, which will sometimes send a SIGKILL to pytest if it's taking too long, and I'm wondering whether this might create a broken state. The only way I've been able to successfully resolve this issue is by running rm -rf .testmondata*.

pokopt commented 1 year ago

Hi! Thanks for feedback. Would you please provide some more details about your setup? Which OS do you use, version of testmon and full command (with all arguments)? Thanks in advance.

christianbundy commented 1 year ago

Yup! I'm running in Docker (based on python:3.10.8-slim-bullseye), my dependencies (all using the latest patch versions) are:

pytest = "~=7.2"
pytest-django = "~=4.5"
pytest-testmon = "~=1.4"
pytest-watcher = "~=0.2.6"
pytest-xdist = "~=3.1"

Pytest options:

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "my_app.settings"
addopts = "-Wignore --no-migrations --reuse-db --testmon"
python_classes = "" # disable pytest-specific class discovery
python_files = ["*_tests.py", "test_*.py", "tests.py"]
python_functions = "" # disable pytest-specific function discovery

My conftest.py:

import pytest

@pytest.fixture(autouse=True)
def enable_db_access_for_all_tests(db):
    pass

And my command:

watchgod pytest.main
christianbundy commented 1 year ago

Logs in case they're useful (one note: I've changed from watchgod to watchfiles, my full command is watchfiles --filter=python pytest):

[17:33:04] watchfiles v0.18.1 👀  path="/home/docker/code" target="bin/pytest" (command) filter=PythonFilter...
================================================================================================================================================ test session starts =================================================================================================================================================
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
django: settings: truework_frontend_api.settings (from env)
testmon: libraries upgrade, changed files: 683, skipping collection of 1739 files, environment: default
rootdir: /home/docker/code, configfile: pyproject.toml
plugins: anyio-3.6.2, requests-mock-1.10.0, postmarker-1.0, Faker-16.6.1, django-4.5.2, testmon-1.4.5, xdist-3.1.0, ddtrace-1.7.3
collecting 5373 items                                                                                                                                                                                                                                                                                                [17:38:33] 1 change detected
[17:38:38] SIGINT timed out after 5 seconds
[17:38:38] process has not terminated, sending SIGKILL
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/_pytest/main.py", line 266, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1037, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/pytest_testmon.py", line 189, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/pytest_testmon.py", line 126, in init_testmon_data
INTERNALERROR>     testmon_data = TestmonData(
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/testmon_core.py", line 139, in __init__
INTERNALERROR>     self.db = db.DB(self.datafile, self.environment)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/db.py", line 49, in __init__
INTERNALERROR>     connection = connect(datafile)
INTERNALERROR>   File "/.venv/lib/python3.10/site-packages/testmon/db.py", line 38, in connect
INTERNALERROR>     connection.execute("PRAGMA synchronous = OFF")
INTERNALERROR> sqlite3.OperationalError: database is locked

I can try increasing my sigkill timeout to see if that makes it happen less often, but it looks to me like maybe the problem is a sigkill while tests are being selected (or something like that)?

christianbundy commented 1 year ago

testmon: libraries upgrade, changed files: 683, skipping collection of 1739 files, environment: default

This could be related -- I'd hit Ctrl+C during a previous test run, but hadn't changed any files. Unsure whether this output is the expected behavior, since not all tests had run, or whether it might be related to this issue.

tarpas commented 1 year ago

I can try increasing my sigkill timeout to see if that makes it happen less often, but it looks to me like maybe the problem is a sigkill while tests are being selected (or something like that)?

What is the filesystem where .testmondata resides? Is it a volume mapped to your development machine? Most likely pytest is killed while the database is locked (testmon is writing) and the operating system doesn't clear the lock (at least not quickly enough) before testmon starts again.. We'll investigate and try to reproduce. I'm googling about stuck file locks in these circumstances but it will take a little longer to investigate.

testmon: libraries upgrade, changed files: 683, skipping collection of 1739 files, environment: default

This could be related -- I'd hit Ctrl+C during a previous test run, but hadn't changed any files. Unsure whether this output is the expected behavior, since not all tests had run, or whether it might be related to this issue.

This is expected and fine.

Thanks again for your time.

christianbundy commented 1 year ago

What is the filesystem where .testmondata resides? Is it a volume mapped to your development machine?

Yep, that's correct. I can try moving it to a non-mapped directory somewhere else in the container, if that's helpful.

tarpas commented 1 year ago

Yep, that's correct. I can try moving it to a non-mapped directory somewhere else in the container, if that's helpful.

Yes, that would be extremely helpful.

And also please, what is you development machine OS/filesystem? We would try to reproduce on the same system.

christianbundy commented 1 year ago

I'm running MacOS 13.0.1 using an encrypted APFS data volume.

axdel commented 1 year ago

Hi, so at this moment we will not tackle this issue, we were not able to reproduce it (tried your exact setup, project with 5k+ tests, watchgod/watchfiles with testmon worked as expected). One thing I'd like to ask you to try, please add:

connection.execute("PRAGMA journal_mode = WAL")

here: https://github.com/tarpas/pytest-testmon/blob/main/testmon/db.py#L37 above other pragma(s), if it helps. WAL mode will be the default in future versions of testmon (but no ETA, sorry).

smitp commented 1 year ago

I am also having the same issue. There is no timeout specified on sqlite3.connect here.

https://github.com/tarpas/pytest-testmon/blob/ef137421d29822ba1b61af5692bd963babe44b06/testmon/db.py#L24

I changed it to 60s and it's not crashing anymore.

    connection = sqlite3.connect(
        f"file:{datafile}{'?mode=ro' if readonly else ''}", uri=True, timeout=60
    )
tarpas commented 12 months ago

@smitp No (long) timeout should be necessary.

Do you use pytest-xdist in the run? Do you run parallel concurrent pytest processes (without xdist?)

tarpas commented 12 months ago

@christianbundy @smitp @DanielSchaffer could you please test with removing the whole line where the exception occurs? connection.execute("PRAGMA synchronous = OFF")

Does it fix the problem? Does the same exception occur on a new place?

Please also (re-)report what kind of parallelism do you use (xdist or something else) and what OS/filesystem do you have.

smitp commented 12 months ago

@tarpas we only use pytest-xdist==3.1.0 to run pytest with multiple workers.

Environment details:

  1. CPU - 16
  2. RAM 64 GB
  3. CI Provider - CircleCI
  4. Runner instance - 2xLarge
  5. xdist -n - 16
  6. Testmon data fle size: ~110MB

could you please test with removing the whole line where the exception occurs? connection.execute("PRAGMA synchronous = OFF")

I tried this but pytest fails with same sqlite3.OperationalError: database is locked error. However, the line that raises OperationalError is different.

> pytest -n16 --testmon
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/xdist/dsession.py", line 115, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/xdist/dsession.py", line 138, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/xdist/dsession.py", line 228, in worker_errordown
INTERNALERROR>     self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw15>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/main.py", line 265, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 982, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 226, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 171, in init_testmon_data
INTERNALERROR>     testmon_data.determine_stable(bool(rpc_proxy))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/testmon_core.py", line 252, in determine_stable
INTERNALERROR>     new_changed_file_data = self.db.fetch_unknown_files(files_fshas, self.exec_id)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/db.py", line 452, in fetch_unknown_files
INTERNALERROR>     con.execute("DELETE FROM changed_files_fshas WHERE exec_id = ?", (exec_id,))
INTERNALERROR> sqlite3.OperationalError: database is locked
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/main.py", line 265, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 982, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 226, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 171, in init_testmon_data
INTERNALERROR>     testmon_data.determine_stable(bool(rpc_proxy))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/testmon_core.py", line 256, in determine_stable
INTERNALERROR>     affected_tests = self.db.determine_tests(self.exec_id, files_mhashes)[
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/db.py", line 484, in determine_tests
INTERNALERROR>     con.execute(
INTERNALERROR> sqlite3.OperationalError: database is locked
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/main.py", line 265, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 982, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 226, in pytest_configure
INTERNALERROR>     init_testmon_data(config)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/pytest_testmon.py", line 171, in init_testmon_data
INTERNALERROR>     testmon_data.determine_stable(bool(rpc_proxy))
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/testmon_core.py", line 252, in determine_stable
INTERNALERROR>     new_changed_file_data = self.db.fetch_unknown_files(files_fshas, self.exec_id)
INTERNALERROR>   File "/home/circleci/Project/venv/lib/python3.9/site-packages/testmon/db.py", line 452, in fetch_unknown_files
INTERNALERROR>     con.execute("DELETE FROM changed_files_fshas WHERE exec_id = ?", (exec_id,))
INTERNALERROR> sqlite3.OperationalError: database is locked
tarpas commented 8 months ago

There is a pre-release ta py pi with https://github.com/tarpas/pytest-testmon/commit/a34500e2ab47a260378ee8a60653ddb7f644ed64 . There is a big hope this should go away.

tarpas commented 8 months ago

@christianbundy @smitp It would be great if you could try the latest pre-release and report if the problem is fixed.

tarpas commented 8 months ago

The processing has been much improved in https://github.com/tarpas/pytest-testmon/releases/tag/v2.1.0 Please report back if it doens't work and you're interested in solution. Thanks.

marcaube commented 7 months ago

I've got the same error when updating pytest-testmon to any version > 2.0.13. Similar setup using pytest-xdist and have 8000-odd tests. Works as soon as I remove -n auto from the command (albeit a lot slower).

tarpas commented 7 months ago

@marcaube please report exact version and the whole stack trace please. It's some kind of deadlock which is not happening on my machine. As said, the handling was much improved and the potential for deadlocks is much smaller.

tarpas commented 7 months ago

@marcaube @smitp @christianbundy @DanielSchaffer this should be fixed now, sorry about the delay. Could anybody test? v2.1.1.dev0 from PYPI?

marcaube commented 7 months ago

Here's a trace just in case, I'll test v2.1.1.dev0 and report back.

============================= test session starts ==============================
platform darwin -- Python 3.11.1, pytest-7.4.3, pluggy-0.13.1
Using --randomly-seed=937020397
testmon: The packages installed in your Python environment have been changed. All tests have to be re-executed. environment: unit
django: version: 3.2.23, settings: settings.test (from option)
rootdir: /Users/marcaube/Projects/backend
configfile: pytest.ini
plugins: ddtrace-1.18.5, Faker-20.0.3, pyfakefs-5.3.1, randomly-3.15.0, xdist-3.4.0, testmon-2.1.0, django-4.7.0, mock-3.12.0, profiling-1.7.0, hypothesis-6.90.0, env-1.1.1
created: 8/8 workers
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> E                 config._do_configure()
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1053, in _do_configure
INTERNALERROR> E                 self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR> E                 res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> E                                                                      ^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 236, in pytest_configure
INTERNALERROR> E                 init_testmon_data(config)
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 173, in init_testmon_data
INTERNALERROR> E                 testmon_data = TestmonData(
INTERNALERROR> E                                ^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/testmon_core.py", line 163, in __init__
INTERNALERROR> E                 result = self.db.initiate_execution(
INTERNALERROR> E                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 693, in initiate_execution
INTERNALERROR> E                 exec_id, packages_changed = self.fetch_or_create_environment(
INTERNALERROR> E                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 660, in fetch_or_create_environment
INTERNALERROR> E                 cursor.execute(
INTERNALERROR> E             sqlite3.OperationalError: attempt to write a readonly database
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .venv/lib/python3.11/site-packages/xdist/dsession.py:200: AssertionError
[gw3] node down: Not properly terminated

replacing crashed worker gw3
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>                                                          ^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 148, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 238, in worker_errordown
INTERNALERROR>     self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw3>

============================= 28 warnings in 8.10s =============================
marcaube commented 7 months ago

Doesn't work for me, even with testmon-2.1.1.dev0 😢 

============================= test session starts ==============================
platform darwin -- Python 3.11.1, pytest-7.4.3, pluggy-0.13.1
Using --randomly-seed=2759008917
testmon: The packages installed in your Python environment have been changed. All tests have to be re-executed. environment: unit
django: version: 3.2.23, settings: settings.test (from option)
rootdir: /Users/marcaube/Projects/backend
configfile: pytest.ini
plugins: ddtrace-1.18.5, Faker-20.0.3, pyfakefs-5.3.1, randomly-3.15.0, xdist-3.4.0, testmon-2.1.1.dev0, django-4.7.0, mock-3.12.0, profiling-1.7.0, hypothesis-6.90.0, env-1.1.1
created: 8/8 workers
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> E                 config._do_configure()
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1053, in _do_configure
INTERNALERROR> E                 self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR> E                 res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> E                                                                      ^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 237, in pytest_configure
INTERNALERROR> E                 init_testmon_data(config)
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 173, in init_testmon_data
INTERNALERROR> E                 testmon_data = TestmonData(
INTERNALERROR> E                                ^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/testmon_core.py", line 163, in __init__
INTERNALERROR> E                 result = self.db.initiate_execution(
INTERNALERROR> E                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 693, in initiate_execution
INTERNALERROR> E                 exec_id, packages_changed = self.fetch_or_create_environment(
INTERNALERROR> E                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 660, in fetch_or_create_environment
INTERNALERROR> E                 cursor.execute(
INTERNALERROR> E             sqlite3.OperationalError: attempt to write a readonly database
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .venv/lib/python3.11/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> E                 config._do_configure()
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1053, in _do_configure
INTERNALERROR> E                 self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR> E                 res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> E                                                                      ^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 237, in pytest_configure
INTERNALERROR> E                 init_testmon_data(config)
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 173, in init_testmon_data
INTERNALERROR> E                 testmon_data = TestmonData(
INTERNALERROR> E                                ^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/testmon_core.py", line 163, in __init__
INTERNALERROR> E                 result = self.db.initiate_execution(
INTERNALERROR> E                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 693, in initiate_execution
INTERNALERROR> E                 exec_id, packages_changed = self.fetch_or_create_environment(
INTERNALERROR> E                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 660, in fetch_or_create_environment
INTERNALERROR> E                 cursor.execute(
INTERNALERROR> E             sqlite3.OperationalError: attempt to write a readonly database
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .venv/lib/python3.11/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> E                 config._do_configure()
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1053, in _do_configure
INTERNALERROR> E                 self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR> E                 res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> E                                                                      ^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> E                 raise ex[1].with_traceback(ex[2])
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 237, in pytest_configure
INTERNALERROR> E                 init_testmon_data(config)
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/pytest_testmon.py", line 173, in init_testmon_data
INTERNALERROR> E                 testmon_data = TestmonData(
INTERNALERROR> E                                ^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/testmon_core.py", line 163, in __init__
INTERNALERROR> E                 result = self.db.initiate_execution(
INTERNALERROR> E                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 693, in initiate_execution
INTERNALERROR> E                 exec_id, packages_changed = self.fetch_or_create_environment(
INTERNALERROR> E                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/testmon/db.py", line 660, in fetch_or_create_environment
INTERNALERROR> E                 cursor.execute(
INTERNALERROR> E             sqlite3.OperationalError: attempt to write a readonly database
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .venv/lib/python3.11/site-packages/xdist/dsession.py:200: AssertionError
[gw0] node down: Not properly terminated

replacing crashed worker gw0
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>                                                          ^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 148, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/marcaube/Projects/backend/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 238, in worker_errordown
INTERNALERROR>     self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw0>

============================= 28 warnings in 8.62s =============================
tarpas commented 7 months ago

@marcaube ah, sorry about that and thanks for your time! I'll work on it.

marcaube commented 7 months ago

@marcaube ah, sorry about that and thanks for your time! I'll work on it.

No problem at all. I appreciate how hard it is to fix a problem that can't easily be reproduced. ❤️

tarpas commented 7 months ago

@marcaube I think I understand better what is happening. Your pytest (xdist) controller process must be collecting different system packages than the worker. That triggers a write operation in the worker (which is not supported). Could you try to run pytest so that poth the controller and the worker have the same system packages? Or maybe you can use undocumented testmon_ignore_packages ini option? https://github.com/tarpas/pytest-testmon/blob/72c70c79aa85095df45bc201adb13bc8e7f7ec96/testmon/pytest_testmon.py#L140C5-L140C71

marcaube commented 7 months ago

Could you try to run pytest so that poth the controller and the worker have the same system packages?

I'm not sure what you mean 🤔 Both pytest and xdist are installed in the project's virtual environment.

tarpas commented 7 months ago

It's quite complicated and therefore I suspect there might be and simpler explanation. Also there are changes coming to testmon where this will not matter, so it's probably not worth investigating this particular issue, I'm sorry.

xdist runs a controller process and worker processes. In the normal execution testmon creates a new "environment" in the controller process and writes it to the .testmondata db. Then the workers just pick that record. In your case for some reason the worker doesn't match the right record and tries to create a new one. I think this function returns something else in the 2 processes. https://github.com/tarpas/pytest-testmon/blob/72c70c79aa85095df45bc201adb13bc8e7f7ec96/testmon/common.py#L36

jasondamour commented 2 months ago

I'm being bit by this too :/ xdist with 8 executors