search

AMICI-dev / AMICI

Advanced Multilanguage Interface to CVODES and IDAS
https://amici.readthedocs.io/
Other
110 stars 31 forks source link

scripts/run-python-tests.sh: line 22: 7262 Segmentation fault #2565

Open dweindl opened 1 month ago

dweindl commented 1 month ago

scripts/run-python-tests.sh: line 22: 7262 Segmentation fault: 11 pytest --ignore-glob=*petab* --ignore-glob=*test_splines.py --durations=10 $@

Happens randomly. So far, twice this week and only on macos. Unclear.

Being updated:

dweindl commented 1 month ago

Another crash, now for python/examples/example_splines_swameye//ExampleSplinesSwameye2003.ipynb on Ubuntu: https://github.com/AMICI-dev/AMICI/actions/runs/11495925466/job/31996499560?pr=2563 :-/

May or may not be related.

dweindl commented 1 month ago

This happens already during import of the amici-extension. I would currently guess that this is related to some external changes (runner image, brew packages, ...)

https://github.com/dweindl/AMICI/actions/runs/11500013663/job/32009264065:

Run scripts/run-python-tests.sh \
Fatal Python error: Segmentation fault

Current thread 0x00000001ef9a0f40 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1233 in create_module
  File "<frozen importlib._bootstrap>", line 573 in module_from_spec
  File "<frozen importlib._bootstrap>", line 676 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/runner/work/AMICI/AMICI/python/sdist/amici/amici.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/runner/work/AMICI/AMICI/python/sdist/amici/__init__.py", line 104 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/runner/work/AMICI/AMICI/python/tests/conftest.py", line 8 in <module>
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1204 in _gcd_import
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126 in import_module
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/pathlib.py", line 582 in import_path
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 710 in _importconftest
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 659 in _loadconftestmodules
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 619 in _try_load_conftest
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 581 in _set_initial_conftests
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1222 in pytest_load_initial_conftests
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1398 in _preparse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1494 in parse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1140 in pytest_cmdline_parse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 341 in _prepareconfig
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 156 in main
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/work/AMICI/AMICI/venv/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
scripts/run-python-tests.sh: line 22: 25340 Segmentation fault: 11  pytest -s --ignore-glob=*petab* --ignore-glob=*test_splines.py --durations=10 $@
Error: Process completed with exit code 139.
dweindl commented 1 month ago

I would currently guess that this is related to some external changes (runner image, brew packages, ...)

Different jobs from the same workflow run:

:hankey:

FFroehlich commented 1 month ago

💩

FFroehlich commented 4 weeks ago

I now get pretty consistent segfaults on the mac workflow in https://github.com/AMICI-dev/AMICI/pull/1861#issuecomment-2439987358

dweindl commented 2 weeks ago

Not quite sure which of those changes could explain those issues -- xcode?

https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20241022.361

https://github.com/actions/runner-images/compare/macos-14-arm64/20241014.301...macos-14-arm64/20241022.361

dweindl commented 2 weeks ago

The changes in the runner-images could just be a coincidence. There also was a new swig release around the same time (https://github.com/swig/swig/releases/tag/v4.3.0).

Recent changes in dependencies: https://ghcr.io/v2/homebrew/core/swig/manifests/4.2.1 -> https://ghcr.io/v2/homebrew/core/swig/manifests/4.3.0 https://ghcr.io/v2/homebrew/core/libomp/manifests/19.1.1 -> https://ghcr.io/v2/homebrew/core/libomp/manifests/19.1.2

further changes: numpy, cmake

dweindl commented 2 weeks ago

It seems to be about swig 4.3.0. I can reproduce that on Ubuntu.

Python crashes specifically when importing amici under pytest. Otherwise it works fine. Interesting...

Update:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
Downloading source file /usr/src/python3.12-3.12.7-1/build-static/../Objects/object.c
0x0000000000584cbe in _PyObject_New (tp=0x0) at ../Objects/object.c:319                                                                                                                                            
warning: 319    ../Objects/object.c: No such file or directory
(gdb) bt
#0  0x0000000000584cbe in _PyObject_New (tp=0x0) at ../Objects/object.c:319
#1  0x00007fffd09cc276 in SWIG_Python_newvarlink () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:1576
#2  0x00007fffd09cc36c in SWIG_globals () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:1606
#3  0x00007fffd0af0dc9 in PyInit__amici () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:85896
#4  0x00000000006a7711 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7ffff414aae0, fp=<optimized out>) at ../Python/importdl.c:169
dweindl commented 2 weeks ago

Python crashes specifically when importing amici under pytest. Otherwise it works fine. Interesting...

It's https://github.com/swig/swig/issues/3061

dweindl commented 2 weeks ago

Leaving this open as a reminder to revert https://github.com/AMICI-dev/AMICI/pull/2572 once the issue has been fixed in swig.

dweindl commented 2 weeks ago

A better fix than https://github.com/AMICI-dev/AMICI/pull/2572#issuecomment-2458929713 might be just ignoring the problematic swig warnings before importing the extension. Untested.