Open rouge8 opened 2 years ago
So I think that ValueError: signal only works in main thread of the main interpreter
is the actually 'interesting' error here. It's funny because lately I've been dealing with this funny business with pytest-xdist myself. I guess if we're in an xdist process then pytest-twisted could choose to not bother with the signals. Though that code was added to avoid twisted taking control of SIGINT
so that pytest stuff could handle it as it wishes. But, if it isn't going to work anyways... I guess maybe we're ok.
https://github.com/pytest-dev/pytest-xdist/issues/620 describes a similar issue with a link to https://github.com/pytest-dev/execnet/issues/96. I commented at 620 so let's see what they say there before we hack around this.
Could you try out #154? I think we may also need to tell Twisted to not configure the signals. You should be able to install it directly such as venv/bin/pip install 'pytest-twisted @ https://github.com/pytest-dev/pytest-twisted@only_set_signals_in_main_thread
.
Hmm, #154 failed on my first run (but looks like it's fine on the second):
❯ pytest -n auto -v src/zg/test/unit/
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
===================================================== test session starts ======================================================
platform darwin -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /Users/andyfreeland/src/pilot/zapgram/.ve/bin/python3.10
cachedir: .pytest_cache
Using --randomly-seed=879619397
rootdir: /Users/andyfreeland/src/pilot/zapgram, configfile: pyproject.toml
plugins: xdist-2.5.0, forked-1.4.0, requests-mock-1.9.3, randomly-3.12.0, unordered-0.5.1, mock-3.8.2, twisted-1.13.4, flakefinder-1.0.0
[gw0] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw1] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw2] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw3] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw4] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw5] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw6] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw7] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
[gw0] Python 3.10.4 (main, Aug 16 2022, 13:53:01) [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw1] Python 3.10.4 (main, Aug 16 2022, 13:53:01) [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw2] Python 3.10.4 (main, Aug 16 2022, 13:53:01) [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw3] Python 3.10.4 (main, Aug 16 2022, 13:53:01) [Clang 13.1.6 (clang-1316.0.21.2.5)]
[gw4] Python 3.10.4 (main, Aug 16 2022, 13:53:01) [Clang 13.1.6 (clang-1316.0.21.2.5)]
gw0 ok / gw1 ok / gw2 ok / gw3 ok / gw4 ok / gw5 C / gw6 C / gw7 CINTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/main.py", line 264, in wrap_session
INTERNALERROR> config._do_configure()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/config/__init__.py", line 995, in _do_configure
INTERNALERROR> self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR> res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 482, in pytest_configure
INTERNALERROR> reactor_installers[config.getoption("reactor")]()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 409, in init_default_reactor
INTERNALERROR> _install_reactor(
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 458, in _install_reactor
INTERNALERROR> init_twisted_greenlet()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 204, in init_twisted_greenlet
INTERNALERROR> signal.signal(
INTERNALERROR> File "/Users/andyfreeland/.asdf/installs/python/3.10.4/lib/python3.10/signal.py", line 56, in signal
INTERNALERROR> handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
INTERNALERROR> ValueError: signal only works in main thread of the main interpreter
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR> """
INTERNALERROR> pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR> pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR> be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR> here ourselves using the formatted message.
INTERNALERROR> """
INTERNALERROR> self._active_nodes.remove(node)
INTERNALERROR> try:
INTERNALERROR> > assert False, formatted_error
INTERNALERROR> E AssertionError: Traceback (most recent call last):
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/main.py", line 264, in wrap_session
INTERNALERROR> E config._do_configure()
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/config/__init__.py", line 995, in _do_configure
INTERNALERROR> E self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR> E res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E return outcome.get_result()
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E raise ex[1].with_traceback(ex[2])
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E res = hook_impl.function(*args)
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 482, in pytest_configure
INTERNALERROR> E reactor_installers[config.getoption("reactor")]()
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 409, in init_default_reactor
INTERNALERROR> E _install_reactor(
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 458, in _install_reactor
INTERNALERROR> E init_twisted_greenlet()
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 204, in init_twisted_greenlet
INTERNALERROR> E signal.signal(
INTERNALERROR> E File "/Users/andyfreeland/.asdf/installs/python/3.10.4/lib/python3.10/signal.py", line 56, in signal
INTERNALERROR> E handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
INTERNALERROR> E ValueError: signal only works in main thread of the main interpreter
INTERNALERROR> E assert False
INTERNALERROR>
INTERNALERROR> .ve/lib/python3.10/site-packages/xdist/dsession.py:192: AssertionError
[gw5] node down: Not properly terminated
replacing crashed worker gw5
[gw8] darwin Python 3.10.4 cwd: /Users/andyfreeland/src/pilot/zapgram
gw0 ok / gw1 ok / gw2 ok / gw3 ok / gw4 ok / gw8 C / gw6 C / gw7 CINTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/xdist/dsession.py", line 117, in pytest_runtestloop
INTERNALERROR> self.loop_once()
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/xdist/dsession.py", line 140, in loop_once
INTERNALERROR> call(**kwargs)
INTERNALERROR> File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/xdist/dsession.py", line 229, in worker_errordown
INTERNALERROR> self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw5>
===================================================== 8 warnings in 13.40s =====================================================
INTERNALERROR> E File "/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py", line 204, in init_twisted_greenlet
INTERNALERROR> E signal.signal(
INTERNALERROR> E File "/Users/andyfreeland/.asdf/installs/python/3.10.4/lib/python3.10/signal.py", line 56, in signal
INTERNALERROR> E handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
INTERNALERROR> E ValueError: signal only works in main thread of the main interpreter
That traceback doesn't agree with the line numbers of the present code in the branch.
Anyways, run it for a bit and let me know how it goes. Pending no issues we can merge etc.
D'oh, it didn't replace the existing pytest-twisted install because the version numbers were the same 🥲. Fixed! I'm not sure of a great way to test it -- I guess I'll try running a subset of our test suite a couple hundred times and see if it errors.
I am sometimes seeing
gw0 ok / gw1 ok / gw2 ok / gw3 ok / gw4 ok / gw5 ok / gw6 C / gw7 ok/Users/andyfreeland/src/pilot/zapgram/.ve/lib/python3.10/site-packages/pytest_twisted.py:205: RuntimeWarning: Failed to block Twisted signal configuration due to not running in the main thread. See https://github.com/pytest-dev/pytest-twisted/issues/153.
I ran it 100 times, got the runtime warning ~25 times, but no crashes!
Thanks. That is an intentional warning. pytest can be set to ignore specific warnings in case you like to make all warnings errors. But, this is a situation that nobody anywhere seems to want to have happening so it seemed reasonable to add the warning. That does confirm though that twisted itself doesn't end up failing out in that scenario, so that is good. I'll plan to get this merged soon. Thanks again for the report and for the testing.
I didn’t notice you merged #154 a couple weeks ago; any plans for a new release?
Should be released in https://pypi.org/project/pytest-twisted/1.14.0/. Sorry for the wait and thanks for the patience.
Thanks!!
We're seeing our test suite occasionally fail to start with
pytest_internalerror() was called on the worker
:I don't have a good reproducer and it's intermittent.
Dependencies:
Running Python 3.10.6 I believe.