search

spcl / dace

DaCe - Data Centric Parallel Programming
http://dace.is/fast
BSD 3-Clause "New" or "Revised" License
487 stars 121 forks source link

FPGA Python interpreter segfaults #1606

Open BenWeber42 opened 2 months ago

BenWeber42 commented 2 months ago

Three FPGA tests cause segfaults in the Python interpreter. (This is based on running FPGA CI with commit https://github.com/spcl/dace/pull/1584/commits/062405addd32889c8a964ad893bfb499187e4757 which disables these tests which results in no more Python segfaults)

The affected tests are:

In the FPGA CI log, they result in the following stack traces:

Fatal Python error: Segmentation fault

Current thread 0x00007fabf0896740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 48 in run_test
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 118 in test_axpy_fpga_array
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
Current thread 0x00007fabf0896740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 48 in run_test
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 124 in test_axpy_fpga_stream
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
Fatal Python error: Segmentation fault

Current thread 0x00007fab4ae49740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/dace/sdfg/sdfg.py", line 2334 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/fpga/veclen_conversion_connector_test.py", line 22 in test_veclen_conversion_connector
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>