Pypy 3.9 segfaults while rewriting test assertions

LilyFoote commented 1 year ago

[x] a detailed description of the bug or problem you are having

When testing the project I work on in CI I frequently see a segfault in pytest's test assert rewriting when running with pypy 3.9. (I haven't yet seen it with 3.10, but I only just added that to CI.)

Run coverage run -m pytest --ignore=tests/simplepoll
  coverage run -m pytest --ignore=tests/simplepoll
  shell: /usr/bin/bash -e {0}
  env:
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    CACHE_ON_FAILURE: false
    pythonLocation: /opt/hostedtoolcache/PyPy/3.9.16/x64/bin
    PYTHONFAULTHANDLER: 1
    RUST_BACKTRACE: 1
Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 346 in _rewrite_test
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 138 in exec_module
  File "<frozen importlib._bootstrap>", line 659 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1002 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1018 in _gcd_import
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/importlib/__init__.py", line 109 in import_module
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/pathlib.py", line 486 in import_path
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 613 in _importtestmodule
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 527 in _getobj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 305 in obj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 536 in _inject_setup_module_fixture
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 530 in collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 372 in <lambda>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 371 in pytest_make_collect_report
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 544 in collect_one_node
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 610 in perform_collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 333 in pytest_collection
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 320 in _main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 257 in wrap_session
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 134 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 182 in console_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 813 in do_run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 611 in command_line
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 964 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/bin/coverage", line 3 in <module>
  File "<builtin>/app_main.py", line 1012 in execfile
  File "<builtin>/app_main.py", line 131 in run_toplevel
  File "<builtin>/app_main.py", line 733 in run_command_line
  File "<builtin>/app_main.py", line 1126 in entry_point
/home/runner/actions-runner/_work/_temp/4b79962b-6f8c-4a60-9e1a-65886acc7eb7.sh: line 1:  1738 Segmentation fault      (core dumped) coverage run -m pytest --ignore=tests/simplepoll
Error: Process completed with exit code 139.

[x] output of pip list from the virtual environment you are using

Package                Version               Editable project location
---------------------- --------------------- --------------------------------------------------
amqp                   5.1.1
anyio                  3.7.0
asgiref                3.6.0
attrs                  23.1.0
billiard               4.1.0
black                  23.3.0
boto3                  1.27.0
botocore               1.30.0
build                  0.10.0
celery                 5.3.1
Cerberus               1.3.4
certifi                2023.5.7
cffi                   1.15.1
charset-normalizer     3.1.0
click                  8.1.3
click-didyoumean       0.3.0
click-plugins          1.1.1
click-repl             0.3.0
colored                1.4.4
coverage               7.2.7
dj-database-url        2.0.0
Django                 3.2.20
django-environ         0.10.0
django-hashid-field    3.3.7
django-pgcrypto-fields 2.6.0
djangorestframework    3.14.0
exceptiongroup         1.1.2
factory-boy            3.2.1
Faker                  18.11.2
freezegun              1.2.2
greenlet               0.4.13
h11                    0.14.0
hashids                1.3.1
hpy                    0.0.4.dev179+g9b5d200
httpcore               0.17.2
httpretty              1.1.4
httpx                  0.24.1
huey                   2.4.5
idna                   3.4
iniconfig              2.0.0
Jinja2                 3.1.2
jmespath               1.0.1
kolo                   2.11.0+local          /home/runner/actions-runner/_work/kolo/kolo/python
kombu                  5.3.1
MarkupSafe             2.1.3
maturin                1.1.0
more-itertools         9.1.0
mypy-extensions        1.0.0
packaging              23.1
pathspec               0.11.1
pip                    23.0.1
platformdirs           3.8.0
pluggy                 1.2.0
prompt-toolkit         3.0.38
pyproject_hooks        1.0.0
pytest                 7.4.0
pytest-asyncio         0.21.0
pytest-celery          0.0.0
pytest-django          4.5.2
pytest-httpx           0.22.0
python-dateutil        2.8.2
pytz                   2023.3
readline               6.2.4.1
requests               2.31.0
s3transfer             0.6.1
setuptools             58.1.0
six                    1.16.0
sniffio                1.3.0
sqlglot                17.0.0
sqlparse               0.4.4
syrupy                 4.0.4
temppathlib            1.2.0
tomli                  2.0.1
tomli_w                1.0.0
toolz                  0.12.0
types-freezegun        1.1.10
types-requests         2.31.0.1
types-urllib3          1.26.25.13
typing_extensions      4.7.1
tzdata                 2023.3
ulid-py                1.1.0
urllib3                1.26.15
vine                   5.0.0
wcwidth                0.2.6

[x] pytest and operating system versions pytest 7.4.0 ubuntu 22.04 (buildjet-2vcpu-ubuntu-2204 https://buildjet.com/for-github-actions/docs/runners/hardware)
[ ] minimal example if possible I'm not sure how to narrow down to the problematic files.

Zac-HD commented 1 year ago

Thanks for the report! I think this is probably a PyPy bug, but let's see if we can narrow it down a bit before we report it upstream.

Since it seems to happen while rewriting assertions, I expect it's possible to trigger this with a single test file, and there's good tooling to analyse what's happening from there.

Do you have a reproducible example? The goal here is a local shell command which reliably triggers the segfault, even if we have to run pytest many times in a loop - debugging when you have to wait for CI each time is... not fun, and being unable to trust your observations is even worse.
Let's try to narrow down which files are implicated. Can you run pytest on (roughly) half of your test suite, and bisect that way? Alternatively, you could patch pytest to print what module it's about to rewrite (with flush=True) and then see what's printed just before the segfault.
Try simplifying the environment and removing dependencies. This is much easier once you have a single file; at that point try deleting chunks of code, imports, etc. with the goal of reproducing in a virtual environment with no proprietary code - that confirms it's an OSS bug somewhere, and lets us try to debug it too.

LilyFoote commented 1 year ago

Sigh, I've spent the whole evening trying to replicate locally and every time the test suite has just passed without triggering the segfault. I've also tried running the github action locally using https://github.com/nektos/act and also gotten nowhere. I'm not sure where to go from here. :disappointed:

Zac-HD commented 1 year ago

Thank you for trying! This is the worst kind of bug, to be clear, https://github.com/HypothesisWorks/hypothesis/issues/3028 got us stuck for several months before we actually worked it out.

Maybe it's dependent on something about the virtualization setup, or on the hardware architecture of your CI runners? Do you know what the physical CPU is on those machines?

The-Compiler commented 1 year ago

Perhaps trying to get something like gdb -ex "set debuginfod enabled on" -ex r -ex bt -ex q --args $(which python) -m pytest to run on GitHub Actions could help, since it hopefully will end up displaying a C stacktrace.

LilyFoote commented 1 year ago

I can now confirm I see this sometimes with pypy 3.10.

@Zac-HD I think the physical cpu is AMD, but I don't know specifics beyond that.

I have also added the gdb command @The-Compiler recommended and I got this:

Run gdb -ex "set debuginfod enabled on" -ex r -ex bt -ex q --args $(which python) -m coverage run -m pytest --ignore=tests/simplepoll
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/python...
Reading symbols from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/pypy3.9.debug...
(No debugging symbols found in /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/pypy3.9.debug)
Starting program: /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/python -m coverage run -m pytest --ignore=tests/simplepoll
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 1955]
[Detaching after fork from child process 1956]

Program received signal SIGBUS, Bus error.
0x00007ffff5a7f9ea in pypy_g_from_object_6 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#0  0x00007ffff5a7f9ea in pypy_g_from_object_6 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#1  0x00007ffff5a9e84d in pypy_g_from_object_59 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#2  0x00007ffff5a800d0 in pypy_g_from_object_6 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#3  0x00007ffff5a81aa0 in pypy_g_from_object_4.part ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#4  0x00007ffff5a8238e in pypy_g_from_object ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#5  0x00007ffff5bc0374 in pypy_g_compile ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#6  0x00007ffff591b050 in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_fsencode_t () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#249 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#250 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#251 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#252 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#253 0x00007ffff632538b in pypy_g_CALL_METHOD_KW__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#254 0x00007ffff5a2e7c5 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#255 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#256 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#257 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#258 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#259 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#260 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#261 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#262 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#263 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#264 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#265 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#266 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#267 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#268 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#269 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#270 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#271 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#272 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#273 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#274 0x00007ffff5bcf9c4 in pypy_g_exec_ ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#275 0x00007ffff575164a in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R_2 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#276 0x00007ffff59a16db in pypy_g_BuiltinCode_funcrun_obj ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#277 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#278 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#279 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#280 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#281 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#282 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#283 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#284 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#285 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#286 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#287 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#288 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#289 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#290 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#291 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#292 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#293 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#294 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#295 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#296 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#297 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#298 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#299 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#300 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#301 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#302 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#303 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#304 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#305 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#306 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#307 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#308 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#309 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#310 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#311 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#312 0x00007ffff5bcf9c4 in pypy_g_exec_ ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#313 0x00007ffff575164a in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R_2 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#314 0x00007ffff59a16db in pypy_g_BuiltinCode_funcrun_obj ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#315 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#316 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#317 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#318 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#319 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#320 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#321 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#322 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#323 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#324 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#325 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#326 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#327 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#328 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#329 0x00007ffff5a2a53d in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#330 0x00007ffff5a3126d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#331 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#332 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#333 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#334 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#335 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#336 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#337 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#338 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#339 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#340 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#341 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#342 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#343 0x00007ffff5a2a53d in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#344 0x00007ffff5a3126d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#345 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#346 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#347 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#348 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#349 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#350 0x00007ffff59a4a45 in pypy_g_call_function.star_3 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#351 0x00007ffff5979959 in pypy_g_entry_point ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#352 0x00007ffff6a1c53f in pypy_main_startup ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#353 0x00007ffff448ed90 in __libc_start_call_main (
    main=main@entry=0x401060 <main>, argc=argc@entry=7, 
    argv=argv@entry=0x7fffffffd7e8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#354 0x00007ffff448ee40 in __libc_start_main_impl (main=0x401060 <main>, 
    argc=7, argv=0x7fffffffd7e8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffd7d8)
    at ../csu/libc-start.c:392
#355 0x000000000040108e in _start ()
A debugging session is active.

    Inferior 1 [process 1952] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

The-Compiler commented 1 year ago

Yeah, that looks like it crashes somewhere deep inside PyPy internals. I'm not really familiar with them, unfortunately. Perhaps it would make more sense to report this there?

LilyFoote commented 1 year ago

I have reported this to the pypy project, with some further investigation of my own: https://foss.heptapod.net/pypy/pypy/-/issues/3960

cfbolz commented 1 year ago

thanks for investigating this, @LilyFoote! that gdb traceback is actually helpful to me! I'm going to try to find out what exactly is crashing.

it's likely something like: pytest assert rewriting produces an AST that is slightly different than what PyPy's ast-to-bytecode compiler expects. and instead of reporting that error sensibly, pypy just segfaults.

cfbolz commented 1 year ago

is there any chance we could try to pretty-print the AST in the CI run that segfaults, before it is passed into compile in the last line of _rewrite_test here?: https://github.com/pytest-dev/pytest/blob/main/src/_pytest/assertion/rewrite.py#L342

cfbolz commented 1 year ago

I trying to get this using a fork of pytest and @LilyFoote's reproducer repo, waiting for results here: https://github.com/cfbolz/pypy-segfault/actions/runs/5485589082/jobs/9994579372

LilyFoote commented 1 year ago

@cfbolz You'll need to enable buildjet: https://buildjet.com/for-github-actions/docs/getting-started/run-your-first-workflow

Thankfully, they have a $5 credit so you don't need to pay until that runs out.

cfbolz commented 1 year ago

ah, I see. they still want a credit card, which I don't have. if I open a PR on your repo with my changes, we could run it on your repo, if that's ok for you @LilyFoote?

cfbolz commented 1 year ago

open one in any case: https://github.com/LilyFoote/pypy-segfault/pull/1

wilhelmklopp commented 1 year ago

(I work with @LilyFoote )

Just in case it's useful to anyone reading through this thread and considering investigating further, Adam from Buildjet did mention the following to us via email:

I'm not sure if our expertise can help with debugging this problem, but we are more than happy to provide free credits to anyone willing to figure it out. Just let me know what account to apply it for, or tell them to message me about, I'll add a large amount of free credits to their account. For context, on the <32vCPU runners, we use AMD 5950x CPUs, and they're accompanied by ECC RAM. Best, Adam

Best email seems to be: support@buildjet.com

olliemath commented 1 year ago

Unsure if this is the same thing, but I'm seeing very similar errors on vanilla EC2 instances. Occasional (but consistent) segfaults - and once also this:

    from base_service.tests.calculator.test_helpers import AllProducts
pypy3-venv/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py:163: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
pypy3-venv/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py:347: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, got <ast.Name object at 0x00000000126c74c0>

which seems to chime with the theory about slightly different ast representations.

I'm using pytest_xdist and pytest_randomly which means the subset of tests executed in each process changes between test runs. I assume this error occurs only when the subset is just right

A5rocks commented 10 months ago

Trio's encountered this in python 3.10 on GitHub Actions with pypy 7.3.13: https://github.com/python-trio/trio/actions/runs/6952653687/job/18916563822. Not sure if this helps anything but this shows it can happen w/ GitHub Actions and with the 3.10 build. I'll see if I can reproduce this in act!

EDIT: unfortunately I don't seem to be able to reproduce this.

cfbolz commented 10 months ago

@A5rocks oh that's great! all my attempts to reproduce locally this or understand what's going on has failed so far. if you found a way to get the problem locally, that would be fantastic.

A5rocks commented 10 months ago

I've reproduced this with a print(ast.dump(tree)). I just don't know which tree might have caused it: https://github.com/A5rocks/trio/actions/runs/7054713178/job/19204406549

cfbolz commented 9 months ago

unfortunately I am still not making any progress on this :-(. My attempts to reproduce this locally have all not worked so far. if I take the dumped ASTs and call rewrite_asserts on them, that works fine. @A5rocks did you manage to get it to fail in act?

A5rocks commented 9 months ago

I realized I was being stupid and that the GH actions run I posted before printed the ast after rewrite_asserts. You can see that ast.dump is in the stack trace, so it's erroring on printing the bad ast.

So I made it print before in https://github.com/A5rocks/trio/actions/runs/7439290913/job/20239062451 but then I tried reproducing locally by running rewrite_asserts on the final AST printed a bunch and then running ast.dump but it didn't segfault. I'm on Windows and didn't feel like setting up WSL at the time though.

mattip commented 8 months ago

have reported this to the pypy project, with some further investigation of my own: https://foss.heptapod.net/pypy/pypy/-/issues/3960

fwiw, PyPy has moved to github, so the upstream issue is now at https://github.com/pypy/pypy/issues/3959

cfbolz commented 6 months ago

just to give a brief update, I am finally making progress on the bug. it's very likely not related to pytest at all, but a bug in pypy's GC(!) that just accidentally seems to manifest in pytest AST rewriting (in several projects). I hope to get a fix merged in the next week or two.

LilyFoote commented 6 months ago

Awesome!! I'd love to see a write up of the problem once you've fixed it.

cfbolz commented 6 months ago

I'd love to see a write up of the problem once you've fixed it.

already working on a draft of that :blush:

LilyFoote commented 6 months ago

This is now fixed upstream: https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html

pytest-dev / pytest

Pypy 3.9 segfaults while rewriting test assertions #11168