search

equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
101 stars 104 forks source link

Investigate ert running slowly in azure #6182

Closed jonathan-eq closed 1 year ago

jonathan-eq commented 1 year ago

Describe the bug It seemed like between release 2023.08.rc3 and rc4, something made ert taking 3x as long to even get the realization boxes showing. We have confirmed the jobs are loaded by checking qstat -fu . Between those release candidates, ert was upgraded from 5.1.0-rc0 to ert 6.0.0-rc1, maybe this is related?

To reproduce Steps to reproduce the behaviour:

jonathan-eq commented 1 year ago

We used git bisect to find the commit in question. It changed pybind for some C code, which could lead to resource starvation. We are suspecting that the reason we only observe the bug in Azure is that the resources there are already sparse as is.