search

equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
103 stars 107 forks source link

Terminate experiment does not work when jobs are pending #7871

Closed berland closed 5 months ago

berland commented 6 months ago

Describe the bug Terminate experiment does not do anything when jobs are pending.

To reproduce Steps to reproduce the behaviour:

  1. Login to an onprem or Azure node.
  2. Run ERT in gui mode (I tried fmu-drogon/drogon_design.ert)
  3. Maybe choose a queue that is very slow to respond, like short, or for LSF use an outrageous memory requirement like QUEUE_OPTION LSF LSF_RESOURCE rusage[mem=200000]
  4. Run an ensemble experiment
  5. While realizations are still in "Pending" state, click "Terminate experiment" and "Yes" in the dialoge box
  6. Observe that nothing happens.
  7. Observe that if the close-button in the upper right corner of the window is clicked, the same dialoge box appears, and if clicking yes, the run-dialogue window is closed. Realizations are still not killed, and the main Ert window has some errors: "Run Experiment" cannot be clicked, and the main window cannot be closed.

Expected behaviour qdel/bkill commands should be initiated and Ert should tear down the ensemble experient

Environment

berland commented 6 months ago

This bug is a regression from https://github.com/equinor/ert/pull/6811

berland commented 6 months ago

The problem is that killing of realizations depends on events being sent, but there are no events while all realizations are pending. The code is stuck in the async for statement in:

https://github.com/equinor/ert/blob/main/src/ert/run_models/base_run_model.py#L502-L532

and never gets to the lines where it would kill realizations.