search

pytest-dev / pytest-xdist

pytest plugin for distributed testing and loop-on-failures testing modes.
https://pytest-xdist.readthedocs.io
MIT License
1.49k stars 232 forks source link

Version 2.0.0 may trigger internal error on Python 3 #580

Open McSinyx opened 4 years ago

McSinyx commented 4 years ago

Hello there, we have just encountered this issue running CIs for pip on Python 3:

============================= test session starts ==============================
platform linux -- Python 3.5.3[pypy-7.0.0-final], pytest-4.6.11, py-1.9.0, pluggy-0.13.1
cachedir: .tox/pypy/.pytest_cache
rootdir: /home/travis/build/pypa/pip, inifile: setup.cfg
plugins: xdist-2.0.0, cov-2.10.1, rerunfailures-8.0, timeout-1.4.2, forked-1.3.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
gw0 [2136]
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/_pytest/main.py", line 206, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/_pytest/main.py", line 250, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/dsession.py", line 112, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/dsession.py", line 135, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/dsession.py", line 238, in worker_collectionfinish
INTERNALERROR>     self.sched.schedule()
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/scheduler/load.py", line 249, in schedule
INTERNALERROR>     self._send_tests(next(nodes), 1)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/scheduler/load.py", line 261, in _send_tests
INTERNALERROR>     node.send_runtest_some(tests_per_node)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/workermanage.py", line 283, in send_runtest_some
INTERNALERROR>     self.sendcommand("runtests", indices=indices)
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/xdist/workermanage.py", line 299, in sendcommand
INTERNALERROR>     self.channel.send((name, kwargs))
INTERNALERROR>   File "/home/travis/build/pypa/pip/.tox/pypy/site-packages/execnet/gateway_base.py", line 728, in send
INTERNALERROR>     raise IOError("cannot send to {!r}".format(self))
INTERNALERROR> OSError: cannot send to <Channel id=1 closed>
======================== no tests ran in 11.77 seconds =========================
ERROR: InvocationError for command /home/travis/build/pypa/pip/.tox/pypy/bin/pytest --timeout 300 --use-venv -m unit -n auto (exited with code 3)

Regarding the error, I found this StackOverflow thread but I'm not sure if it is still applicable to current pytest-xdist.

nicoddemus commented 4 years ago

Hi @McSinyx,

Unfortunately that's a generic "something bad happened while sending tests to the worker", and has a large number of possible causes (some of them not even related to pytest-xdist). Can you provide a reproducible example?

McSinyx commented 4 years ago

Thank you for the prompt response. As for reproducibility, I can't make the error appear on my machine, but the CIs consistently show the error since last night and pytest-xdist happens to have a new release recently. I thought that the ambiguous errors above would contain some obvious information to the pytest team but I guess my imagination is too wild :smile:

To be fair pip is using a quite obsolete combination of version for pytest and its plugins and pinning pytest-xdist to 1.34.0 appears to solve the issue. I don't speak for the maintainers of pip but I think it is not urgent at all to have the root cause discovered/fixed in the next few months (until pip drops Python 2 support).

Edit: Using latest pytest (instead of <5) also makes the CIs green. Please feel free to edit the title or close this issue, whichever you think is better.

tylerjereddy commented 4 years ago

I'm seeing this consistently in SciPy Azure CI since the latest pytest-xdist release and when using pytest==5.4.3. The latter is pinned because of https://github.com/pytest-dev/pytest/issues/7592.

This is observed on Windows only I think.

tylerjereddy commented 4 years ago

See PR here for example: https://github.com/scipy/scipy/pull/12667

Note that those logs can disappear pretty quickly

tylerjereddy commented 4 years ago

@nicoddemus Looks pretty reproducible for SciPy, but replicating our exact CI environment may take devs a fair bit of time on Windows. Forking SciPy repo and setting up Azure CI on the fork is what I do in these cases.

almightyfoon commented 4 years ago

This issue looks like its an issue in pytest 5.4.3, upgrading to 6.0.1 fixed the issue for my team.

kdelee commented 4 years ago

Confirming this is is happening to us for some older maintenance releases of ours that are pinned on pytest<6.0.0

Have reproduced on fedora 32 and centos8 and rhel8, so no special platform issue. These are all using python 3.6.

For those maintenance releases we've pinned pytest-xdist to 1.34.0 and now are avoiding problem.

Is there some way to warn that pytest-xdist>2.0.0 requires pytest>=6.0.0 ?

Not a problem on unpinned pytest and pytest-xdist in our devel branch.

alimcmaster1 commented 4 years ago

Is there some way to warn that pytest-xdist>2.0.0 requires pytest>=6.0.0 ?

setup.py should handle this: https://github.com/pytest-dev/pytest-xdist/blob/master/setup.py#L3