search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.34k stars 1.14k forks source link

[good first issue][CI breakage] `pytest-xdist == 3.6.0` makes the CI fail #4317

Closed webknjaz closed 2 weeks ago

webknjaz commented 3 weeks ago

Since the release of new pytest-xdist roughly an hour ago, all CI jobs fail as follows:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/main.py", line 273, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/main.py", line 327, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 166, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 138, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/xdist/dsession.py", line 201, in worker_workerfinished
INTERNALERROR>     self.config.hook.pytest_testnodedown(node=node, error=None)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pytest_cov/plugin.py", line 316, in pytest_testnodedown
INTERNALERROR>     self.cov_controller.testnodedown(node, error)
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/pytest_cov/engine.py", line 342, in testnodedown
INTERNALERROR>     rinfo = node.gateway._rinfo()
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/execnet/gateway.py", line 87, in _rinfo
INTERNALERROR>     self._cache_rinfo = RInfo(ch.receive())
INTERNALERROR>                               ^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/setuptools/setuptools/.tox/py/lib/python3.11/site-packages/execnet/gateway_base.py", line 934, in receive
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> execnet.gateway_base.RemoteError: concurrent remote_exec would cause deadlock for main_thread_only execmodel

(https://github.com/pypa/setuptools/actions/runs/8759251151/job/24041851385?pr=4092#step:9:114)

I also filed the upstream report here: https://github.com/pytest-dev/pytest-xdist/issues/1071.

Immediate action item

~Temporarily restrict the version of this dependency by adding , != 3.6.0 to the dependency specifier in tox.ini and maybe elsewhere, this needs investigation.~ It looks like execnet is what needs to be capped below v2 per https://github.com/pytest-dev/pytest-xdist/issues/620#issuecomment-2067333447.

MorganWeiss commented 3 weeks ago

I am experiencing the same issue as well. Is there a way of changing the version to have it work? I do not see a dependency in my packages so I am not sure where it is coming from.

abravalheri commented 3 weeks ago

If I understood correctly, as per https://github.com/pytest-dev/pytest-xdist/issues/1071#issuecomment-2067565494 pytest==3.6.0 has been yanked, and probably new versions of pytest-xdist, execnet and pytest will be coordinated to fix the problem, right?

So my understanding is that for the time being we don't have to pin any dependency as a workaround. (Also happy to add a workaround if I understood things wrong).

webknjaz commented 3 weeks ago

@abravalheri I haven't tested it but I'd expect that it's execnet that needs to be yanked, not pytest-xdist. So my guess is that it's still possible to get a broken version. You can restart any GHA workflow in the repo and see if it breaks..

abravalheri commented 3 weeks ago

@abravalheri I haven't tested it but I'd expect that it's execnet that needs to be yanked, not pytest-xdist. So my guess is that it's still possible to get a broken version. You can restart any GHA workflow in the repo and see if it breaks..

Possibly working? (https://github.com/pypa/setuptools/actions/runs/8759517556/job/24055273503, although cygwin seems to have trouble on finding the dependencies).

webknjaz commented 3 weeks ago

Possibly working? (https://github.com/pypa/setuptools/actions/runs/8759517556/job/24055273503

Yep

although cygwin seems to have trouble on finding the dependencies).

No, that's an incorrect assessment. This is a classical case of "who knows what broke in the CI step" — if you look closer, you'll see that pip install was successful and coverage xml is what actually failed. This is one of the reasons I always insist on having a single command per CI step (it actually enables more confusing behaviors when there's several invocations inside, like a "false success").

MorganWeiss commented 3 weeks ago

Any progress on fixing this issue?

abravalheri commented 3 weeks ago

@MorganWeiss I don't see any immediate action to be taken by setuptools. Maybe it is relevant for you to follow the issues in pytes-xdist and co and possibly contributing towards the fix there?

MorganWeiss commented 3 weeks ago

I am not aware of those, do you have any helpful links?

Get Outlook for iOShttps://aka.ms/o0ukef

From: Anderson Bravalheri @.> Sent: Saturday, April 20, 2024 11:33:31 PM To: pypa/setuptools @.> Cc: Morgan Weiss @.>; Mention @.> Subject: Re: [pypa/setuptools] [good first issue][CI breakage] pytest-xdist == 3.6.0 makes the CI fail (Issue #4317)

@MorganWeisshttps://github.com/MorganWeiss I don't see any immediate action to be taken by setuptools. Maybe it is relevant for you to follow the issues in pytest, pytes-xdist and co?

— Reply to this email directly, view it on GitHubhttps://github.com/pypa/setuptools/issues/4317#issuecomment-2067924865, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKH4GZL5ILPLPFJ4UG5JPSDY6NMTXAVCNFSM6AAAAABGP3UTMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXHEZDIOBWGU. You are receiving this because you were mentioned.Message ID: @.***>

abravalheri commented 3 weeks ago

You can start with the link shared in the first message of this thread, and then you can follow the discussions (they point to more links there).

webknjaz commented 2 weeks ago

v3.6.1 has just been released. Hopefully, this is enough to address this problem.