equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
101 stars 104 forks source link

Adaptive Localization on Drogon fails due to lack of memory #6926

Closed dafeda closed 8 months ago

dafeda commented 8 months ago

Describe the bug Exception in thread ert_gui_simulation_thread: Traceback (most recent call last): File "/opt/rh/rh-python38/root/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/opt/rh/rh-python38/root/usr/lib64/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/gui/simulation/run_dialog.py", line 273, in run self._run_model.startSimulations( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/run_models/base_run_model.py", line 249, in startSimulations run_context = self.run_experiment( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/run_models/ensemble_smoother.py", line 123, in run_experiment smoother_snapshot = smoother_update( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/analysis/_es_update.py", line 982, in smoother_update analysis_ES( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/analysis/_es_update.py", line 679, in analysis_ES ] = smoother_adaptive_es.assimilate( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/iterative_ensemble_smoother/experimental.py", line 264, in assimilate corr_XY = self._correlation_matrix(cov_XY, X, Y) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/iterative_ensemble_smoother/experimental.py", line 155, in _correlation_matrix corr_XY: npt.NDArray[np.double] = (cov_XY / stds_X[:, np.newaxis]) / stds_Y[ numpy.core._exceptions.MemoryError: Unable to allocate 44.8 GiB for an array with shape (886512, 6790) and data type float64

dafeda commented 8 months ago

Solved by https://github.com/equinor/iterative_ensemble_smoother/commit/1ef022dc20ff30fbe0cd4543211a6861787ce0bc