search

equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
101 stars 105 forks source link

tests causes `Fatal Python error: Aborted` sometimes #8157

Closed eivindjahren closed 2 months ago

eivindjahren commented 3 months ago

https://github.com/equinor/ert/actions/runs/9499211555/job/26179690419

The following has been seen in komodo tests:

QThread: Destroyed while thread is still running
Fatal Python error: Aborted

Thread 0x00007fc021175700 (most recent call first):
  <no Python frame>

Thread 0x00007fbf8d7ea700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File "/usr/lib64/python3.11/threading.py", line 622 in wait
  File ".../site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fc070d04700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File ".../site-packages/opentelemetry/sdk/_logs/_internal/export/__init__.py", line 249 in worker
  File "/usr/lib64/python3.11/threading.py", line 975 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fc071505700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File "/usr/lib64/python3.11/threading.py", line 622 in wait
  File ".../site-packages/azure/monitor/opentelemetry/exporter/_utils.py", line 195 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fc0c41b2700 (most recent call first):
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 534 in read
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 567 in from_io
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 341 in run
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Current thread 0x00007fc0d43b0740 (most recent call first):
  File ".../site-packages/_pytest/runner.py", line 141 in runtestprotocol
  File ".../site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File ".../site-packages/pluggy/_callers.py", line 103 in _multicall
  File ".../site-packages/pluggy/_manager.py", line 120 in _hookexec
  File ".../site-packages/pluggy/_hooks.py", line 513 in __call__
  File ".../site-packages/xdist/remote.py", line 195 in run_one_test
  File ".../site-packages/xdist/remote.py", line 174 in pytest_runtestloop
  File ".../site-packages/pluggy/_callers.py", line 103 in _multicall
  File ".../site-packages/pluggy/_manager.py", line 120 in _hookexec
  File ".../site-packages/pluggy/_hooks.py", line 513 in __call__
  File ".../site-packages/_pytest/main.py", line 339 in _main
  File ".../site-packages/_pytest/main.py", line 285 in wrap_session
  File ".../site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File ".../site-packages/pluggy/_callers.py", line 103 in _multicall
  File ".../site-packages/pluggy/_manager.py", line 120 in _hookexec
  File ".../site-packages/pluggy/_hooks.py", line 513 in __call__
  File ".../test-kenv/root/lib64/python3.11/site-packages/xdist/remote.py", line 393 in <module>
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1291 in executetask
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 341 in run
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1273 in serve
  File ".../test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1806 in serve
eivindjahren commented 2 months ago

Seems like this was solved by #8318 as we haven't seen this failure in a while. We can reopen if we see it again