equinor / xtgeo

XTGeo Python class library for subsurface Surfaces, Cubes, Wells, Grids, Points, etc
https://xtgeo.readthedocs.io/en/latest/
GNU Lesser General Public License v3.0
110 stars 56 forks source link

DRAFT: Lazy well trajectory #1102

Closed janbjorge closed 4 months ago

janbjorge commented 8 months ago

Addresses #997

This pull request details an optimization overhaul in the well import logic. The update is quantified by benchmarking comparisons, highlighting a significant reduction in data loading times.

The improvement represents around a 40x speed increase, noticeable only when trajectory data is not required. The load times for individual wells have decreased markedly, with each well's load time reduced from a span of 0.097 to 0.438 seconds (old) to a range of 0.003 to 0.013 seconds (new).

Drogon well import (old)

Well load: 55_33-1 - 0.29210895812138915
Well load: 55_33-2 - 0.3175274571403861
Well load: 55_33-3 - 0.2579669498372823
Well load: 55_33-A-1 - 0.2322694668546319
Well load: 55_33-A-2 - 0.2290720408782363
Well load: 55_33-A-3 - 0.22684309305623174
Well load: 55_33-A-4 - 0.4384316320065409
Well load: 55_33-A-5 - 0.31261010793969035
Well load: 55_33-A-6 - 0.16252269502729177
Well load: RFT_55_33-A-2 - 0.09771290514618158
Well load: RFT_55_33-A-3 - 0.10098752984777093
Well load: RFT_55_33-A-4 - 0.23368731187656522
Well load: RFT_55_33-A-5 - 0.10333754192106426
Well load: RFT_55_33-A-6 - 0.09616
Wells load 3.103423219872638

Drogon well import (new)

Well load: 55_33-1 - 0.0068743119481951
Well load: 55_33-2 - 0.004892189987003803
Well load: 55_33-3 - 0.0049566240049898624
Well load: 55_33-A-1 - 0.0051729329861700535
Well load: 55_33-A-2 - 0.005157941021025181
Well load: 55_33-A-3 - 0.0053442229982465506
Well load: 55_33-A-4 - 0.012952441116794944
Well load: 55_33-A-5 - 0.005436931969597936
Well load: 55_33-A-6 - 0.005160709144547582
Well load: RFT_55_33-A-2 - 0.003282841993495822
Well load: RFT_55_33-A-3 - 0.003077584085986018
Well load: RFT_55_33-A-4 - 0.004737856099382043
Well load: RFT_55_33-A-5 - 0.0030709300190210342
Well load: RFT_55_33-A-6 - 0.0030067998450249434
Wells load 0.07474512211047113

40x speedup (if you do not ask for the trajeceties)

Tested on Troll, we see the same factor of on loading performce ~40x (8.5 min eager, 12s with lazy-loading).

TODO: Lazy loading should bring the load time close to zero. Need to figure out where the extra overhead is coming from.

codecov-commenter commented 8 months ago

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (03624f7) 80.14% compared to head (b50b74b) 80.15%.

Files Patch % Lines
src/xtgeo/common/pandas_extensions.py 85.45% 7 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1102 +/- ## ======================================= Coverage 80.14% 80.15% ======================================= Files 96 97 +1 Lines 13575 13630 +55 Branches 2193 2197 +4 ======================================= + Hits 10880 10925 +45 - Misses 1963 1971 +8 - Partials 732 734 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.