Running tests fails with "node down: Not properly terminated", maybe execnet related?

jenyacazacu commented 6 years ago

We recently moved to Python 3.4 and when running our tests with pytest-xdist installed and in parallel with (-n) sometimes the tests fail with the below error. The traceback is from using faulthandler. The tests consistently fail during tests that use celery tasks and that have the format:

@override_settings(task_always_eager=True) def test_example(self):

Packages used: execnet-1.5.0 pytest-3.0.6 xdist-1.15.0

Here is the error log:

platform linux -- Python 3.4.5, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 Django settings: gameserver.settings.test (from environment variable) rootdir: /home/jcazacu/Repos/main_repo/ares-game-server, inifile: tox.ini plugins: xdist-1.15.0, faulthandler-1.4.1, django-3.1.2, cov-2.5.1, celery-4.0.2 gw0 [535] / gw1 [535] / gw2 [535] / gw3 [535] scheduling tests via LoadScheduling .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Fatal Python error: Segmentation fault

Thread 0x00007fbd16d3d700 (most recent call first): File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 386 in read File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 418 in from_io File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Current thread 0x00007fbd1eed2740 (most recent call first): File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 335 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/utils.py", line 62 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 288 in _savepoint_rollback File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 328 in savepoint_rollback File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/transaction.py", line 243 in exit File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1004 in _rollback_atomics File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1066 in _fixture_teardown File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 908 in _post_teardown File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/channels/tests/base.py", line 57 in _post_teardown File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 216 in call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/unittest.py", line 157 in runtest File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 104 in pytest_runtest_call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 163 in init File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in call_runtest_hook File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 133 in call_and_report File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 79 in runtestprotocol File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call File "", line 77 in run_tests File "", line 61 in pytest_runtestloop File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 133 in _main File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 98 in wrap_session File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 127 in pytest_cmdline_main File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call File "", line 159 in File "", line 1 in do_exec File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1072 in executetask File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 259 in integrate_as_primary_thread File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1047 in serve File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1534 in serve File "", line 8 in File "", line 1 in ......[gw1] node down: Not properly terminated fReplacing crashed slave gw1

Any ideas?

Currently we just moved the task tests to run serially since the failures are so unpredictable.

RonnyPfannschmidt commented 6 years ago

its really not clear whats happening there, its just clear that python itself gets seg-faulted, which is quite a feat and a real pain to debug

jenyacazacu commented 6 years ago

ok, thx, do you have any suggestions as to where I can start with debugging?

RonnyPfannschmidt commented 6 years ago

unfortunately not, im not familiar with the surrounding libs in use

kydbakep commented 4 years ago

[gw4] [ 96%] PASSED tests/warehouse/test_warehouse_supplier_refunds.py::TestDeleteSupplierRefund::test_delete_refund_after_delete_cell 10:32:56 [gw5] [ 96%] PASSED tests/warehouse/test_warehouse_supplier_refunds.py::TestDeleteSupplierRefund::test_delete_refund_after_good_has_become_serial 10:32:56 [gw3] node down: Not properly terminated 10:32:56 [gw3] [ 96%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_close_residue_rule_via_mask 10:32:56

10:32:56 replacing crashed worker gw3 10:32:56

10:32:56 [gw8] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9 10:32:57

10:32:57 [gw8] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0] 10:33:00

10:33:10 tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_positive 10:33:10 [gw2] node down: Not properly terminated 10:33:10 [gw2] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[Abc] 10:33:10

10:33:10 replacing crashed worker gw2 10:33:10

10:33:10 [gw9] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9 10:33:10

10:33:10 [gw9] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0] 10:33:13

10:33:16 tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[ ] 10:33:16 [gw7] node down: Not properly terminated 10:33:16 [gw7] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestCreateAndDeleteResidueControl::test_check_table_icon 10:33:16

10:33:16 replacing crashed worker gw7 10:33:16

10:33:16 [gw10] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9 10:33:17

10:33:17 [gw10] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0] 10:33:20

10:34:01 tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_field_validation_negative[0] 10:34:01 [gw0] node down: Not properly terminated 10:34:01 [gw0] [ 97%] FAILED tests/warehouse/test_warehouse_residue.py::TestResidueControlValidation::test_close_residue_rule_modal 10:34:01

10:34:01 replacing crashed worker gw0 10:34:01

10:34:01 [gw11] linux Python 3.7.6 cwd: /opt/buildagent/work/61408fc2f9cf70c9 10:34:02

10:34:02 [gw11] Python 3.7.6 (default, Mar 17 2020, 13:08:12) -- [GCC 7.5.0]

espears1 commented 3 years ago

I have seen this type of error occur when combining pytest-xdist with other libraries that deal with multithreaded or multi-processing code. One common pain point is the Annoy library from Spotify, especially older versions. It does many multi-threaded actions without end user control of the threading, and there are just extremely mystrerious and un-debuggable ways that pytest-xdist causes some bad interaction with threading to lead to a segfault and a crashed worker. Worse this can be different for different OSes or Python versions.

I am debugging an example right now where on Mac OS with Python 3.6.9, everything is fine and pytest-xdist runs my test suite (with parallel workers) without issue. If I take the same code and put it in an equivalent Ubuntu Docker image and run the tests, I get a segfault from execnet and crashed workers. But if I run the tests serially in the Docker image, all tests pass.

These issues are virtually impossible to boil down to simplified reproducible examples as well, since if I knew what all the factors were that are required to minimally reproduce it, that would probably be the solution to debugging it. It's very, very hard to use pytest-xdist in these situations.

brechtm commented 2 years ago

I'm seeing a similar error (Python segfault + "node down: Not properly terminated"). It seems to fail consistently in my Ubuntu CI runs (these logs will be unavailable soon) and on my MacBook, but only on CPython 3.11 (3.11.0-beta.3 on Ubuntu and 3.11.0b3+ [f9d0240] on macOS).

nox > python run_tests.py -m 'not longrunning' --script-launch-mode=subprocess -s tests_regression

``` ============================= test session starts ============================== platform linux -- Python 3.11.0b3, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/runner/work/rinohtype/rinohtype, configfile: setup.cfg plugins: assume-2.4.3, github-actions-annotate-failures-0.1.6, console-scripts-1.3.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0 gw0 I / gw1 I gw0 [88] / gw1 [88] .................../home/runner/work/rinohtype/rinohtype/tests_regression/rst/columns_two.rst:1: (ERROR/3) Undefined substitution referenced: "problematic". ....../home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:11: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. /home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:18: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. /home/runner/work/rinohtype/rinohtype/tests_regression/rst/exceptional_style.rst:21: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent. ............../home/runner/work/rinohtype/rinohtype/tests_regression/rst/inline_markup.rst:1: (ERROR/3) Undefined substitution referenced: "problematic". .................Fatal Python error: Segmentation fault Thread 0x00007f92f6d6f700 (most recent call first): File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 400 in read File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 432 in from_io File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 220 in run File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn Current thread 0x00007f92f7854740 (most recent call first): File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 466 in File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 466 in _index File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 473 in get_rowspanned_columns File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 562 in __int__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 575 in __iter__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 537 in __eq__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 303 in match File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 361 in match File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 622 in match File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 739 in find_matches File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 742 in find_matches File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 742 in find_matches File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 327 in get_matches File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 697 in _get_value_lookup File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/attribute.py", line 469 in get_value_for File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/attribute.py", line 333 in get_config_value File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/style.py", line 512 in get_style File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/util.py", line 160 in function_wrapper File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/draw.py", line 86 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 386 in draw_cell_border File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 401 in _place_rows_and_render_borders File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 166 in render_rows File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/table.py", line 176 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 581 in _flow_with_next File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 572 in _flow_with_next File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 535 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 295 in flow_inner File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/flowable.py", line 240 in flow File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 637 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 366 in _render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 306 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 197 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/layout.py", line 197 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 178 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/template.py", line 461 in render File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 478 in _render_pages File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/rinoh/document.py", line 436 in render File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 105 in render_doctree File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 92 in _render_file File "/home/runner/work/rinohtype/rinohtype/tests_regression/helpers/regression.py", line 59 in render_rst_file File "/home/runner/work/rinohtype/rinohtype/tests_regression/test_rst.py", line 45 in test_rst File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/python.py", line 1761 in runtest File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 259 in File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 130 in runtestprotocol File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 110 in run_one_test File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 91 in pytest_runtestloop File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/xdist/remote.py", line 291 in File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1084 in executetask File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 220 in run File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1060 in serve File "/home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/lib/python3.11/site-packages/execnet/gateway_base.py", line 1554 in serve File "", line 8 in File "", line 1 in Extension modules: markupsafe._speedups (total: 1) ...........[gw1] node down: Not properly terminated F replacing crashed worker gw1 s................... =================================== FAILURES =================================== _________________________ tests_regression/test_rst.py _________________________ [gw1] linux -- Python 3.11.0 /home/runner/work/rinohtype/rinohtype/.nox/regression-3-11-sdist/bin/python worker 'gw1' crashed while running 'tests_regression/test_rst.py::test_rst[png]' =============================== warnings summary =============================== .nox/regression-3-11-sdist/lib/python3.11/site-packages/babel/messages/catalog.py:13 .nox/regression-3-11-sdist/lib/python3.11/site-packages/babel/messages/catalog.py:13 ... ```

I have seen this type of error occur when combining pytest-xdist with other libraries that deal with multithreaded or multi-processing code.

All tests perform image comparison in parallel (diffpdf.py), but the crash does not occur in this stage of the test but rather during rinohtype's rendering which is single-threaded.

It's always _testrst[png] that segfaults. When running without pytest-xdist (-n 0), all is well. It is possible to reproduce the issue running only the _testrst[png] test case:

git clone https://github.com/brechtm/rinohtype.git
cd rinohtype
git checkout bd4b4157
poetry install
poetry run nox -r -s "regression-3.11(wheel)" -- -k png

pytest (7.1.2)
pytest-xdist (2.5.0)

UPDATE: Running the tests not through nox, the issue doesn't occur!

.nox/regression-3-11-wheel/bin/python run_tests.py -m 'not longrunning' --script-launch-mode=subprocess -k png tests_regression

link2xt commented 1 year ago

EDIT: this turned out to be a bug in our code, not pytest: https://github.com/deltachat/deltachat-core-rust/pull/4153

WenjieDu commented 1 year ago

Confronted with the same problem in PyPOTS testing here https://github.com/WenjieDu/PyPOTS/actions/runs/4577128474/jobs/8082161160

basnijholt commented 1 year ago

No solution, but for others that every come across this issue, in my case the following happened.

When running pytest tests in parallel with xdist, my tests crash with:

[gw5] node down: Not properly terminated
[gw5] FAILED tests/test.py::test_with_botorch

replacing crashed worker gw5

This crash happens only when I have:

class A:
    def x(self):
        import botorch # only used in this function

but not when

import botorch

class A:
    def x(self):

Also cannot reproduce it on MacOS but only in the CI on DevOps with ubuntu-latest.

gonchik commented 1 year ago

@basnijholt
Just curious, do you still detecting the same construction? did you played with number of threads etc?

pytest-dev / pytest

Running tests fails with "node down: Not properly terminated", maybe execnet related? #3216