equinor / ert

ERT - Ensemble based Reservoir Tool - is designed for running ensembles of dynamical models such as reservoir models, in order to do sensitivity analysis and data assimilation. ERT supports data assimilation using the Ensemble Smoother (ES), Ensemble Smoother with Multiple Data Assimilation (ES-MDA) and Iterative Ensemble Smoother (IES).
https://ert.readthedocs.io/en/latest/
GNU General Public License v3.0
99 stars 104 forks source link

Cache realization states to avoid redundant checking of files #8135

Open yngve-sk opened 1 month ago

yngve-sk commented 1 month ago

(this is a draft that shows somewhat what it would look like if we added state monitoring, currently there is quite some room for refactoring/improvement in readability/more tests)

Example of the state:

self.to_dataframe()
                                                      value     timestamp
realization kind                key                                      
0           SNAKE_OIL_PARAM_BPR BPR_138_PERSISTENCE    True  1.718870e+09
                                BPR_555_PERSISTENCE    True  1.718870e+09
            SNAKE_OIL_PARAM_OP1 OP1_DIVERGENCE_SCALE   True  1.718870e+09
                                OP1_OCTAVES            True  1.718870e+09
                                OP1_OFFSET             True  1.718870e+09
                                OP1_PERSISTENCE        True  1.718870e+09
            SNAKE_OIL_PARAM_OP2 OP2_DIVERGENCE_SCALE   True  1.718870e+09
                                OP2_OCTAVES            True  1.718870e+09
                                OP2_OFFSET             True  1.718870e+09
                                OP2_PERSISTENCE        True  1.718870e+09
            TOP                 TOP                    True  1.718870e+09

TOP->TOP indicates that the parameter TOP was updated (no groups here), kind->key otherwise indicates RESPONSE TYPE -> RESPONSE KEY / PARAMETER GROUP -> PARAMETER KEY

Benchmarking (Testing on RGS on large plot case)

Issue Resolves #8125

codecov-commenter commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 96.10778% with 13 lines in your changes missing coverage. Please review.

Project coverage is 86.97%. Comparing base (7281a93) to head (15fe79f).

Files Patch % Lines
src/ert/storage/realization_state.py 93.64% 11 Missing :warning:
src/ert/storage/local_ensemble.py 98.72% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #8135 +/- ## ========================================== + Coverage 86.86% 86.97% +0.10% ========================================== Files 379 380 +1 Lines 23971 24293 +322 Branches 620 624 +4 ========================================== + Hits 20822 21128 +306 - Misses 3068 3086 +18 + Partials 81 79 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.