pytest-xdist interop: signal only works in main thread

[pytest-timeout-bot]

Original report by Buck Evan (Bitbucket: bukzor, GitHub: bukzor).

---

I'm getting this kind of error with increasing frequency as I add tests to my project.

#!python

INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 59, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 72, in run_tests
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 70, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 95, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

Full detail:

• http://paste.pound-python.org/show/qnK7qab1vZEYbYIzwLXD/
• http://paste.pound-python.org/show/5SnhQMlELjJgiE3Lwpim/

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

Oops, apologies to miss the original report!

It seems like execnet must have changed where xdist's DSession is run and it is no longer run in the main thread. Not immediately sure what to do about this but I think you should be able to work around this by using --timeout-method=thread.

[pytest-timeout-bot]

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

---

Any update on this issue? hitting this regularly

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

@karthikborkar does the work-around fix it for you?

[pytest-timeout-bot]

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

---

@flub From https://pypi.python.org/pypi/pytest-timeout the thread method has few downsides such as relatively large overhead for running each test and that test runs are not completed. This means that other py.test features, like e.g. JUnit XML output or fixture teardown, will not function normally. fixture teardown and JUnit XML are important features which i need them to work correctly

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

But it is the most reliable way of using pytest-timeout. pytest-timeout aims to be a last-resort attempt at finding problems that you otherwise would just be stuck with and not get any useful infor from. It's kind of misleading to think you can still continue your test run after a timeout occurred. And I do wonder if the default method is wrong because of this.

[pytest-timeout-bot]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

---

I am getting the same kind of errors when using pytest-timeout with pytest-xdist:

https://travis-ci.org/Peque/osbrain/jobs/197634358

Using --timeout-method=thread fixes this issue (i.e.: test suite completes) but, when a timeout actually occurs, the whole test suite gets stuck (just as if I didn't use pytest-timeout) and no useful stack-trace information is given. The output does not tell me which test timed-out either.

[pytest-timeout-bot]

Original comment by Bulat Gaifullin (Bitbucket: bgaifullin, GitHub: bgaifullin).

---

the issue affects me too.

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

So currently it seems pytest-timeout and pytest-xdist seem to work together fine, I ran this:

import pytest, time

@pytest.mark.parametrize('a', [1, 2, 3, 4])
@pytest.mark.timeout(1)
def test_foo(a):
    time.sleep(2)
    assert a

And the timeouts work. So I'll close this unless someone reopens with a reproducible example.

Thanks

[pytest-timeout-bot]

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

---

Can still reproduce this. With pytest < 3.3 it would happen nearly every run, now it happens every fifth run or so.

#!python
from time import sleep

import pytest

@pytest.mark.timeout(10000)
@pytest.mark.parametrize("value", range(0, 20))
def test_timeout(value):
    sleep(5)
    pass

Full output:

#!python

(venv) darius@darius-ubuntu:~/dev/repro$ pytest test_timeout.py -v -n4
============================================================================================================================ test session starts ============================================================================================================================
platform linux2 -- Python 2.7.12, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /home/darius/dev/repro/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/darius/dev/repro, inifile:
plugins: xdist-1.22.2, timeout-1.2.1, forked-0.2
[gw0] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw1] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw2] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw3] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw0] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw3] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw1] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw2] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
gw0 [20] / gw1 [20] / gw2 [20] / gw3 [20]
scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session

test_timeout.py::test_timeout[1] 
test_timeout.py::test_timeout[3] 
test_timeout.py::test_timeout[2] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 65, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 84, in run_one_test
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 175, in _multicall
INTERNALERROR>     next(gen)   # first yield
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 76, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 115, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 116, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 139, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 181, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_timeout.py::test_timeout[0]', <WorkerController gw2>)
INTERNALERROR> assert not 'test_timeout.py::test_timeout[0]'

======================================================================================================================= no tests ran in 5.64 seconds ========================================================================================================================

Keep in mind, I'm not actually triggering the timeout, it's much larger than the test can reasonably take. Runs fine without -n4.

pip list:

#!python
Package        Version
-------------- -------
apipkg         1.4    
attrs          18.1.0 
execnet        1.5.0  
funcsigs       1.0.2  
more-itertools 4.1.0  
pip            10.0.1 
pluggy         0.6.0  
py             1.5.3  
pytest         3.5.1  
pytest-forked  0.2    
pytest-timeout 1.2.1  
pytest-xdist   1.22.2 
setuptools     39.1.0 
six            1.11.0 
wheel          0.31.1 

Running on Ubuntu 16.04. I have not noticed this issue on Windows or Mac.

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

Sorry, with your sample file in a loop I still can't reproduce this. I did a pip install pytest pytest-timeout pytest-xdist. I even added pytest-forked as maybe a trigger but running the test in a loop still didn't trigger anything.

Have you tried this in a complete clean environment? Or is there anything else hidden, like a config file or so (as otherwise pytest-forked wouldn't do anything)

[pytest-timeout-bot]

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

---

Ok, so I tried redoing everything from scratch, these are my steps:

#!python
virtualenv venv
source venv/bin/activate
pip install pytest pytest-timeout pytest-xdist
pytest test_timeout.py -v -n4
pytest test_timeout.py -v -n4

It failed on the second run. I also checked which pytest just in case and it's the one in my virtual environment.

I'm not sure what other system information could be relevant. I'm seeing this both on my Ubuntu 16.04 VM (running in Parallels on a Mac) and on our Jenkins servers, also Ubuntu 16.04.

[pytest-timeout-bot]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

---

I too think this should be reopened. See:

https://travis-ci.org/opensistemas-hub/osbrain/jobs/379614919

With:

• pytest==3.5.1
• pytest-timeout==1.2.1
• pytest-xdist==1.22.2

That is the result of trying to reintegrate pytest-timeout in osBrain with no other changes.

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

I am not sure that pytest-timeout has anything to do with this directly.

FWIW I am getting the same kind of errors in scancode-toolkit test runs... but ONLY on Travis/Linux and not always.... kinda like every other run like others have reported here... and re-running the failed run makes the issue go away.

See https://travis-ci.org/nexB/scancode-toolkit/jobs/458977684#L1189

The tests I run are using pytest and xdist AND the underlying code supports timeout-based interruption in multiprocessing . See https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py#L121

It could well be that the code I used for timeouts is using the same calls that pytest-timeout... which could explain why I am facing this too. In any case, I thought posting here could help

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

ok, my timeout handling code is functionally the same set of calls that pytest-timeout uses. So it makes sense to experience the same issues.

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

Now the interesting part is that pytest-xdist does not seem to be part of the problem either as Darius said above https://bitbucket.org/pytest-dev/pytest-timeout/issues/8/pytest-xdist-interop-signal-only-works-in#comment-45439827

So this seems to boil down t an issue between pytest and timout signals

[lucrib]

Any news about this? I am facing this issue. I am trying to gather new logs for this.

[lucrib]

Not sure if this will help or not, but I found some python bugs related to this: https://bugs.python.org/issue38904 https://bugs.python.org/issue39042 Also this thread on stack overflow. https://stackoverflow.com/questions/54749342/valueerror-signal-only-works-in-main-thread

I don't have knowledge enough to figure out how to fix this, but if someone could give me some direction I could try to submit a PR for this.

[uniqueni]

I am facing this issue too and I have no workaround please

[flub]

I've re-read this whole thread again. Most of these complaints seem to me like some other environments which are making pytest run in not the main thread and/or other signal handling from the application interfering. I'm going to close this again as I don't think this is overly useful currently.

[mchlnix]

I encountered this problem also with the combination of a CI, xdist and timeout. Signals crashed regularly and threads made the test run stuck at 100%.

So in the script my CI worker was executing, I set xdist to use all, but one core. This left one core/thread idle, which I hoped could be used by the OS for whatever, thus ensuring, that all the test-runners are undisturbed and can all claim their own main thread.

It has worked so far.

For linux: CPU_COUNT="$(grep -c "^processor" /proc/cpuinfo)" so pytest - n $(( CPU_COUNT - 1 ))

EDIT: It dramatically reduced the frequency, but the error does still occur.

[ndevenish]

I know this is an old thread, and appreciate the unhelpfulness of "me too" reports, but we just ran into this several-CI-runs-in-a-row, with basically the same stacktrace as the original (Jenkins on platform linux -- Python 3.8.6, pytest-6.2.1, py-1.10.0, pluggy-0.13.1/; plugins: dials-data-2.1.36, timeout-1.4.2, xdist-2.2.0, mock-3.5.1, forked-1.2.0).

If it's just happening for tests that have timed out then it's not much of a problem, it would have failed anyway. But the stacktrace looks like it's a problem registering the handler, rather than executing it?

I seem to be able to quite reliably reproduce with a clean environment via Docker:

from ubuntu:latest

RUN apt update && apt install -y python3 python3-pip && pip3 install pytest pytest-xdist pytest-timeout
RUN printf '\
from time import sleep\n\
import pytest\n\
@pytest.mark.timeout(10000)\n\
@pytest.mark.parametrize("value", range(0, 20))\n\
def test_timeout(value):\n\
    sleep(5)\n\
    pass' >> t.py
RUN echo "set -e; while true; do pytest t.py -v -n20; done" > dotest.sh

This seems to reproduce it pretty reliably, every time:

$ docker build -t testtimeout .
$ docker run -it testtimeout bash dotest.sh
....
INTERNALERROR> E             ValueError: signal only works in main thread
INTERNALERROR> E           assert False

Where -n >> number of cores (my docker VM on this machine only has one core allocated to it). I wonder, if it's something to do with long time spent collecting?

[ndevenish]

Some more notes:

• pytest t.py -v -n20 on it's own seems to trigger the problem ~80% of the time
• I cannot get this to reproduce without -v verbose mode - removing -v and it happily runs over and over, at least as long as I tried
• I inserted this code in before the signal.signal line:

  pathlib.Path(f"{os.getpid()}-{threading.current_thread().ident}.log").write_text(f"""
  {"GOOD" if threading.current_thread() == threading.main_thread() else "BAD"}
  {threading.current_thread().ident=}
  {threading.main_thread().ident=}
  {os.getpid()=}
  {"".join(traceback.format_stack())}
  """)

  and the tracebacks are identical for good & bad except the bad instances don't have the stack before _perform_spawn e.g.:

  
  File "<string>", line 1, in <module>
  File "<string>", line 8, in <module>
  File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 1554, in serve
  SlaveGateway(io=io, id=id, _startcount=2).serve()
  File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 1060, in serve
  self._execpool.integrate_as_primary_thread()
  File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 267, in integrate_as_primary_thread
  self._perform_spawn(reply)

^^^^ Traceback above here does not appear on "BAD" runs where thread != main_thread

File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 285, in _perform_spawn reply.run() File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 220, in run self._result = func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/execnet/gateway_base.py", line 1084, in executetask do_exec(co, loc) # noqa .... File "/usr/local/lib/python3.8/dist-packages/pytest_timeout.py", line 104, in pytest_runtest_protocol timeout_setup(item) File "/usr/local/lib/python3.8/dist-packages/pytest_timeout.py", line 213, in timeout_setup {"".join(traceback.format_stack())}

which, reading https://github.com/pytest-dev/execnet/blob/master/execnet/gateway_base.py#L267 - seems to indicate that sometimes this is called explicitly as a primary thread, and in some combination of cases I'm not yet certain on, execnet spawns a worker as a non-primary-thread.

I'm not sure yet if this is a problem with `pytest-timeout`'s assumption that it's called on _a_ main thread, or whether there is some race-condition bug in execnet/pytest-xdist

[ndevenish]

So, https://github.com/ndevenish/pytest-timeout/commit/bf39d0a68053f6ef3fcaa90a4ee5c248862af1c6 fixes this issue, in the most trivial way - it falls back to thread mode if it detects that signal is going to fail. I'm not sure the full balance of choice between thread and signal, I've read in https://github.com/postlund/pyatv/issues/904 that thread mode has an increased risk of hanging - but hopefully since it'd only be falling back to thread mode in the corner cases, I think this is a small enough risk for my use cases.

If this was the route to take - and I suspect it's significantly simpler than finding the cause in xdist/execnet - then maybe a third "auto" mode (or debug "signalwithfallback") could be added and explicitly used by the people who seem to be running into this more often.

I'm not sure if this thread is monitored so if not, I'll make a PR to start the conversation at some point.

[flub]

Hi @ndevenish, thanks for the explorations and additional info! The commit you point to is very interesting indeed. I'm a bit surprised that with xdist threaded mode can hang, I that'd be interesting to understand better as well.

Certainly do take this to a PR, at first glance I'm a bit torn on automatically switching to threaded vs just erroring out and forcing someone to decide this explicitly. But it would be nice to improve usability somehow and you found a pretty promising way I think.

(also, apologies I'm not always the most active, responses can take a while as you just noticed)

[ndevenish]

(also, apologies I'm not always the most active, responses can take a while as you just noticed)

I have no expectations for reply times for open source projects! I'll try to pull a PR together, for discussion if nothing else.

[graingert]

@flub this should be fixed (mostly) on execnet==1.8.0

see also https://github.com/pytest-dev/pytest-xdist/issues/620

[ndevenish]

Ah, that's good to hear. I was settling on "Error, but with a more helpful suggestion" as the least-worst way to "fix", and didn't want to dig into execnet. Evidently it annoyed someone further than it annoyed me :)

[flub]

@graingert oh, that's very nice work you did! Might finally fix this issue properly and would mean we don't need a workaround here.

[pseudotensor]

Yes, we've gotten ever since using xdist + timeout together. It's fairly random:

internal error
def worker_internal_error(self, node, formatted_error):
        """
        pytest_internalerror() was called on the worker.

        pytest_internalerror() arguments are an excinfo and an excrepr, which can't
        be serialized, so we go with a poor man's solution of raising an exception
        here ourselves using the formatted message.
        """
        self._active_nodes.remove(node)
        try:
>           assert False, formatted_error
E           AssertionError: Traceback (most recent call last):
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/_pytest/main.py", line 257, in wrap_session
E                 session.exitstatus = doit(config, session) or 0
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/_pytest/main.py", line 313, in _main
E                 config.hook.pytest_runtestloop(session=session)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
E                 return self._hookexec(self, self.get_hookimpls(), kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
E                 return self._inner_hookexec(hook, methods, kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
E                 return outcome.get_result()
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
E                 raise ex[1].with_traceback(ex[2])
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
E                 res = hook_impl.function(*args)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/remote.py", line 75, in pytest_runtestloop
E                 self.run_one_test(torun)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/remote.py", line 89, in run_one_test
E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
E                 return self._hookexec(self, self.get_hookimpls(), kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
E                 return self._inner_hookexec(hook, methods, kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
E                 return outcome.get_result()
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
E                 raise ex[1].with_traceback(ex[2])
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 182, in _multicall
E                 next(gen)  # first yield
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pytest_timeout.py", line 104, in pytest_runtest_protocol
E                 timeout_setup(item)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pytest_timeout.py", line 207, in timeout_setup
E                 signal.signal(signal.SIGALRM, handler)
E               File "/opt/h2oai/dai/python/lib/python3.8/signal.py", line 47, in signal
E                 handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
E             ValueError: signal only works in main thread
E           assert False

/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/dsession.py:187: AssertionError

[xoviat]

@flub Can you close this?

[flub]

I guess so, pytest-timeout now recognises this situation and will refuse to use the signal timeout method, falling back to the thread method.