equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
99 stars 104 forks source link

unit_tests/job_queue/test_workflow.py::test_workflow failed nightly tests #8150

Open sondreso opened 1 month ago

sondreso commented 1 month ago

Describe the bug

unit_tests/job_queue/test_workflow.py::test_workflow 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Could not call mock_job_script due to No such file or directory
Fatal Python error: Aborted

Thread 0x00007fad55ffb700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad56ffd700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad57fff700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad6d5fc700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad6e5fe700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad9b08d700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad99f8b700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_job_queue_node.py", line 98 in write_to_queue
  File "/usr/lib64/python3.11/threading.py", line 1394 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad6f4ff700 (most recent call first):
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib64/python3.11/threading.py", line 975 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad9a88c700 (most recent call first):
  File "/usr/lib64/python3.11/selectors.py", line 468 in select
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 1884 in _run_once
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 607 in run_forever
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 640 in run_until_complete
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_ert_forward_model_runner/client.py", line 130 in send
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_ert_forward_model_runner/reporting/event.py", line 108 in _event_publisher
  File "/usr/lib64/python3.11/threading.py", line 975 in run
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_ert/threading.py", line 47 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fad027e4700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File "/usr/lib64/python3.11/threading.py", line 622 in wait
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fade6567700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/opentelemetry/sdk/_logs/_internal/export/__init__.py", line 249 in worker
  File "/usr/lib64/python3.11/threading.py", line 975 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fade6d68700 (most recent call first):
  File "/usr/lib64/python3.11/threading.py", line 324 in wait
  File "/usr/lib64/python3.11/threading.py", line 622 in wait
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/azure/monitor/opentelemetry/exporter/_utils.py", line 195 in run
  File "/usr/lib64/python3.11/threading.py", line 1038 in _bootstrap_inner
  File "/usr/lib64/python3.11/threading.py", line 995 in _bootstrap

Thread 0x00007fae39c12700 (most recent call first):
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 534 in read
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 567 in from_io
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 341 in run
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Thread 0x00007fae49e10740 (most recent call first):
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/parsers/earley_forest.py", line 366 in visit
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/parsers/earley_forest.py", line 538 in visit
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/parsers/earley_forest.py", line 401 in transform
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/parsers/earley.py", line 305 in parse
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/parser_frontends.py", line 104 in parse
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/lark/lark.py", line 658 in parse
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/ert/config/parsing/lark_parser.py", line 415 in _parse_file
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/ert/config/parsing/lark_parser.py", line 490 in parse
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/ert/config/workflow.py", line 40 in _parse_command_list
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/ert/config/workflow.py", line 99 in from_file
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test_root/tests/unit_tests/job_queue/test_workflow.py", line 35 in test_workflow
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/python.py", line 1632 in runtest
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/xdist/remote.py", line 195 in run_one_test
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/xdist/remote.py", line 174 in pytest_runtestloop
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/main.py", line 339 in _main
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/prog/res/komodo/bleeding-py311-rhel8/root/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/xdist/remote.py", line 393 in <module>
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1291 in executetask
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 341 in run
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1273 in serve
  File "/tmp/f_scout_ci/actions-runner-02/_temp/test-kenv/root/lib64/python3.11/site-packages/execnet/gateway_base.py", line 1806 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

[gw3] node down: Not properly terminated
[gw3] FAILED unit_tests/job_queue/test_workflow.py::test_workflow 

Full log: https://github.com/equinor/komodo-releases/actions/runs/9508607090/job/26210860253

To reproduce Steps to reproduce the behaviour:

  1. TBD

Expected behaviour Not crash

Screenshots N/A

Environment

Additional context N/A