search

pytest-dev / pytest-forked

extracted --boxed from pytest-xdist to ensure backward compat
MIT License
62 stars 21 forks source link

crashing test hangs `pytest` #55

Closed stefanseefeld closed 2 years ago

stefanseefeld commented 2 years ago

I'm observing the pytest process itself seemingly breaking as a result of a segmentation fault observed in a test. Here is the relevant part of the log:

../python/tests/ctk/simulator/test_simple.py::test_battery Fatal Python error: Segmentation fault

Current thread 0x00007f412136e700 (most recent call first):
<no Python frame>

Thread 0x00007f4181d75780 (most recent call first):
  File "/home/user/.local/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 76 in _child
  File "/home/user/.local/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 50 in __init__
  File "/home/user/.local/lib/python3.8/site-packages/pytest_forked/__init__.py", line 67 in forked_run_report
  File "/home/user/.local/lib/python3.8/site-packages/pytest_forked/__init__.py", line 46 in pytest_runtest_protocol
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/user/.local/bin/py.test", line 8 in <module>
Fatal Python error: Aborted

Current thread 0x00007f412136e700 (most recent call first):
<no Python frame>

Thread 0x00007f4181d75780 (most recent call first):
  File "/home/user/.local/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 76 in _child
  File "/home/user/.local/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 50 in __init__
  File "/home/user/.local/lib/python3.8/site-packages/pytest_forked/__init__.py", line 67 in forked_run_report
  File "/home/user/.local/lib/python3.8/site-packages/pytest_forked/__init__.py", line 46 in pytest_runtest_protocol
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/user/.local/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/home/user/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/user/.local/bin/py.test", line 8 in <module>

Any idea how this might happen ? Is this a pytest bug ?

stefanseefeld commented 2 years ago

I just had a realization: The pytest-forked code runs the actual test in a sub-process, but not the fixtures, correct ? In my code the python-wrapped C++ application that is being tested is created and destroyed within fixtures, and I have strong suspicions that the crash actually happens in the teardown phase. Assuming all of the above is true, how hard is it to run fixtures (at least all the ones that run per test rather than per session) within the forked process as well ?

nicoddemus commented 2 years ago

That makes sense, but we try to run the entire setup/call/teardown in a subprocess:

https://github.com/pytest-dev/pytest-forked/blob/759912b731b6ba227dc274421f3be534184574bd/src/pytest_forked/__init__.py#L53-L75

Not sure how that plays with the fixture cache though.

stefanseefeld commented 2 years ago

...and I have yet to be able to build a minimal test case where a segfault in either setup, test, or teardown results in an error similar to what I reported earlier.

stefanseefeld commented 2 years ago

I have been able to trace this back to an incorrectly written signal handler in my own code. Sorry for the noise !

RonnyPfannschmidt commented 2 years ago

@stefanseefeld thanks for the followup