Error message with title "Request Failed" when trying to plot non-summary vectors from AHM-case together with a PRED-case

[KetilN]

Description

An error message pops up with title "Request Failed" when trying to plot AHM-case together with a PRED-case as shown in screenshot below.

Note that when plotting summary vectors, no error message pops up.

To reproduce I am running /prog/res/komodo/2023.02.05-py38-rhel7/root/bin/ert

1. Copy the FMU Drogon-project version 21.0.3.
2. Run ert test_run drogon_ahm.ert
3. Run ert test_run drogon_pred_ref.ert
4. Run ert gui drogon_pred_ref.ert and open the "Create Plot"-window.
5. Choose the default_0 case which is a single-run-case in the ERT storage.
6. Try to plot a vector like NPV@0 which is a non-summary vector.
7. See that "big error message window pops up".
8. Check that you can click ok on the error message and continue using the plotter.

Environment

• OS: RHEL7
• ERT/Komodo release: 2023.02.05-py38-rhel7
• Python version: 3.8
• Remote/HPC execution involved: [yes|no] : both To reproduce the data, HPC is relevant. To get the error, only interactive usage is relevant.

Additional context So the issue seems to be related to having both AHM and Prediction case in the ERT Storage at the same time, and then trying to plot non-summary data from AHM case.

[andreas-el]

@KetilN You indicate that you are running 2023.02.05, but the screenshot provided seems to show 2022.12.06. Is there a mixup somehow here?

[yngve-sk]

@KetilN is there a special reason for including step 2 and 3? I managed to recreate this error going straight to step4.

ert gui drogon_pred_ref.ert

then doing the steps you outlined:

4. Run ert gui drogon_pred_ref.ert and open the "Create Plot"-window.
5. Choose the default_0 case which is a single-run-case in the ERT storage.
6. Try to plot a vector like NPV@0 which is a non-summary vector.

[KetilN]

@KetilN You indicate that you are running 2023.02.05, but the screenshot provided seems to show 2022.12.06. Is there a mixup somehow here?

Well, you are right that there is an in-consistent screenshot here. I think the point is that I tested this both for 2022.12.06 and for 2023.02.05 and I got the error in both the versions.

[KetilN]

@KetilN is there a special reason for including step 2 and 3? I managed to recreate this error going straight to step4.

ert gui drogon_pred_ref.ert

then doing the steps you outlined: 4. Run ert gui drogon_pred_ref.ert and open the "Create Plot"-window. 5. Choose the default_0 case which is a single-run-case in the ERT storage. 6. Try to plot a vector like NPV@0 which is a non-summary vector.

Tja, well, doing the steps listed is example on a relevant user procedure. But it is still very valuable information from you that this error occur directly on FMU Drogon-project version 21.0.3.

So, maybe you (or your team) can suggest if this error message is expected in this situation when running towards such an ERT-project with later ERT-version like 2022.12.06 and 2023.02.05, and that we then need to use an older ERT-version to be able to open this ERT-project ? Follow-up question would then be: Is it possible to see which version of ERT that was used to store the data in the ERT-storage for a given ERT-project ?

[andreas-el]

@KetilN is there a special reason for including step 2 and 3? I managed to recreate this error going straight to step4.

ert gui drogon_pred_ref.ert

then doing the steps you outlined: 4. Run ert gui drogon_pred_ref.ert and open the "Create Plot"-window. 5. Choose the default_0 case which is a single-run-case in the ERT storage. 6. Try to plot a vector like NPV@0 which is a non-summary vector.

Tja, well, doing the steps listed is example on a relevant user procedure. But it is still very valuable information from you that this error occur directly on FMU Drogon-project version 21.0.3.

So, maybe you (or your team) can suggest if this error message is expected in this situation when running towards such an ERT-project with later ERT-version like 2022.12.06 and 2023.02.05, and that we then need to use an older ERT-version to be able to open this ERT-project ? Follow-up question would then be: Is it possible to see which version of ERT that was used to store the data in the ERT-storage for a given ERT-project ?

First off, i know that dependencies FastApi and Starlette were updated between komodo-versions 2022.12 and 2023.02

-starlette: 0.19.1
+starlette: 0.20.4
-fastapi: 0.79.0
+fastapi: 0.85.0

I'm mentioning this due to the fact that these are mentioned repeatedly in the error message you've provided. So there might be some issues that could arise if mixing these versions.

I'm going to assume that this procedure worked at some point, and that it is expected to be able to do this in more recent versions. So the next obvious steps would be to try to determine a komodo version where this actually works A long shot would be to try to run this with an older version of Drogon just in case.

[andreas-el]

I can reproduce the issue using komodo-version 2022.12, but on 2022.11 and 2022.10 the NPV@0 does not contain plotable data at all (no error message though).

@yngve-sk found that running the drogon_ahm.ert in the 21.0.3 version of drogon fails due to this issue;

Traceback (most recent call last):
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/bin/ecl2csv", line 8, in <module>
    sys.exit(main())
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/lib/python3.8/site-packages/ecl2df/ecl2csv.py", line 315, in main
    args.func(args)
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/lib/python3.8/site-packages/ecl2df/ecl2csv.py", line 306, in run_subparser_main
    main_func(args)
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/lib/python3.8/site-packages/ecl2df/summary.py", line 922, in summary_main
    write_dframe_stdout_file(sum_df, args.output, index=True, caller_logger=logger)
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/lib/python3.8/site-packages/ecl2df/common.py", line 136, in write_dframe_stdout_file
    pyarrow.feather.write_feather(dframe, dest=output)
  File "/prog/res/komodo/2022.10.05-py38-rhel7/root/lib64/python3.8/site-packages/pyarrow/feather.py", line 195, in write_feather
    _feather.write_feather(table, dest, compression=compression,
  File "pyarrow/_feather.pyx", line 41, in pyarrow._feather.write_feather
  File "pyarrow/io.pxi", line 1716, in pyarrow.lib.get_writer
  File "pyarrow/io.pxi", line 945, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 960, in pyarrow.lib.OSFile._open_writable
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 113, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file 'share/results/unsmry/DROGON-0.arrow'. Detail: [errno 2] No such file or directory

I suspect the data found in the ert/output/storage folder found in the drogon dataset does not contain valid data currently. We are trying to resolve the aforementioned issue to see if a specific komodo-release cause the originally reported problem.

[andreas-el]

Opened tickets in fmu-drogon: https://github.com/equinor/fmu-drogon/issues/121 https://github.com/equinor/fmu-drogon/issues/120 https://github.com/equinor/fmu-drogon/issues/122

[KetilN]

Thanks a lot for very good follow-up of this topic.
We test now this functionality again with the current "komodo stable" version - and the current "komodo testing" version, and if it now works ok, this is fine for now.

However, in the long run, it is needed to get a proper answer on this question:

Is it possible to see which version of ERT that was used to store the data in the ERT-storage for a given ERT-project ?

[KetilN]

Sorry to report, but still get this error in the "komodo stable" version:

How the popup error window was produced:

0. Use "komodo stable today" and using latest Drogon version (only internal in FMU-Domain-team right now. Will soon be internal available in Equinor).
1. Start with an "empty storage folder".
2. Run ert gui drogon_ahm.ert --- Single Realization
3. Run ert gui drogon_pred_ref.ert --- - First add and change the default case to pred_ref.. Then run Single Realization.
4. Then click on datatype = "NPV@0" and then get the "Request Failed" popup window.

[KetilN]

Some more detailed info:

The error message occurs when having chosen the "default case" (also called AHM-case) in the QT Plotter-window.

And then this error message is printed in the Terminal window where "ert gui drogon_pred_ref.ert" is run:

2023-03-20 17:55:48,816 [ERROR] uvicorn.error: Exception in ASGI application Traceback (most recent call last): File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi result = await app( # type: ignore[func-returns-value] File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call return await self.app(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/fastapi/applications.py", line 271, in call await super().call(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/applications.py", line 118, in call await self.middleware_stack(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 706, in call await route.handle(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 66, in app response = await func(request) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib/python3.8/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(**values) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib64/python3.8/site-packages/ert/dark_storage/endpoints/records.py", line 178, in get_ensemble_record dataframe = data_for_key(res, ensemble_name, name, realization_index) File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib64/python3.8/site-packages/ert/dark_storage/common.py", line 56, in data_for_key data = res.gather_gen_data_data(case, key, realization_index).T File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib64/python3.8/site-packages/ert/libres_facade.py", line 534, in gather_gen_data_data data = self.load_gen_data( File "/prog/res/komodo/2023.03.01-py38-rhel7/root/lib64/python3.8/site-packages/ert/libres_facade.py", line 228, in load_gen_data data_array = _clib.enkf_fs_general_data.gendata_get_realizations( IndexError: _Map_base::at

[KetilN]

When the "plot case = pref_ref" is chosen, no error message pops up, the Plotter window works as expected, ref:

[KetilN]

Tested this now in "komodo bleeding". Slightly better in "komodo bleeding", but still able to get the popup error message, ref:

Below is the error message in the Terminal Window.

Will retest this using "komodo testing" when "komodo testing" is different from "komodo stable".

---

2023-03-21 00:02:04,676 [ERROR] uvicorn.error: Exception in ASGI application Traceback (most recent call last): File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/xarray/core/dataset.py", line 1348, in _construct_dataarray variable = self._variables[name] KeyError: 'ECALC'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi result = await app( # type: ignore[func-returns-value] File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call return await self.app(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/fastapi/applications.py", line 271, in call await super().call(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/applications.py", line 118, in call await self.middleware_stack(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 706, in call await route.handle(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/starlette/routing.py", line 66, in app response = await func(request) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(**values) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/dark_storage/endpoints/records.py", line 179, in get_ensemble_record dataframe = data_for_key(res, db.get_ensemble(ensemble_id), name, realization_index) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/dark_storage/common.py", line 52, in data_for_key data = res.gather_gen_kw_data(case, key, realization_index) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/libres_facade.py", line 374, in gather_gen_kw_data data = self.load_all_gen_kw_data( File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/libres_facade.py", line 359, in load_all_gen_kw_data gen_kw_dict = fs.load_gen_kw_as_dict(key, realization) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/storage/local_ensemble.py", line 216, in load_gen_kw_as_dict data, keys = self.load_gen_kw_realization(key, realization) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib64/python3.8/site-packages/ert/storage/local_ensemble.py", line 294, in load_gen_kw_realization np_data = ds_disk.sel(iens=realization)[key].to_numpy() File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/xarray/core/dataset.py", line 1439, in getitem return self._construct_dataarray(key) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/xarray/core/dataset.py", line 1350, in _constructdataarray , name, variable = _get_virtual_variable(self._variables, name, self.dims) File "/prog/res/komodo/bleeding-py38-rhel7/root/lib/python3.8/site-packages/xarray/core/dataset.py", line 186, in _get_virtual_variable raise KeyError(key) KeyError: 'ECALC'

[KetilN]

IMPROVEMENT OBSERVED: New test today in "komodo bleeeding" does not give the popup error message when choosing trying to plot non-summary vectors in a combined AHM/PRED-case. Ref. screen capture below.

Can you confirm that the code that handles plotting vectors in the QT plotter has been adjusted in bleeding last few days ?

[andreas-el]

I know there were some bugs ironed out related to storage yesterday on bleeding. There is also new versions of stable present, meaning that some high-value issues were resolved.

Thanks for the extended re-runs of this issue. I assume bleeding still does not satisfy, since there should have been some plots visible on NPV@0.

Regarding the version of storage; Yes. There is a hidden .fs_version file present in the storage directory. Mine says version 0 still, so i don't think there will any immediate value there.

[KetilN]

THANK YOU VERY MUCH. My first impression now is that you have fixed the bugs that I have reported.

[KetilN]

Note that I think the "komodo bleeding" version now works fine related to NPV@0 since there probably never have been any data to plot there (in the AHM-cases called default. default_0 and default_1) - it is just that we do not want the end-users to see the "HUGE ERROR POPUP MESSAGES".

[KetilN]

Before we "close this bug report":

The .fs_version file concept is PERFECT ! It will be very good to have - especially related to "later issues" for end-users that want to re-open old ERT-storage areas.

Can you please send a complete path/listing of the hidden .fs_version file ? I do not see it in "my bleeding version", see output below.

kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm % ls -al storage/.fs_version ls: cannot access storage/.fs_version: No such file or directory kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm % ls -al storage/ensembles/ total 192 drwxrwsr-x 6 kenu fmu 216 Mar 21 09:54 . drwxrwsr-x 3 kenu fmu 27 Mar 21 10:53 .. drwxrwsr-x 102 kenu fmu 3277 Mar 21 00:45 26a5e739-2d2b-448e-b1aa-405b637222ae drwxrwsr-x 3 kenu fmu 56 Mar 21 00:21 51e96f74-5b4e-4e55-8844-b0a24ccad155 drwxrwsr-x 102 kenu fmu 3277 Mar 21 00:44 9241d56c-7acd-4b12-8961-5147667897af drwxrwsr-x 103 kenu fmu 3282 Mar 21 09:54 d1ec9d3d-dba6-4a8d-b2fc-931f4f819eb7 kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm % ls -al storage/ensembles/.fs_version ls: cannot access storage/ensembles/.fs_version: No such file or directory kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm % ls -al storage/ensembles/d1ec9d3d-dba6-4a8d-b2fc-931f4f819eb7/.fs_version ls: cannot access storage/ensembles/d1ec9d3d-dba6-4a8d-b2fc-931f4f819eb7/.fs_version: No such file or directory kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm % ls -al storage/ensembles/d1ec9d3d-dba6-4a8d-b2fc-931f4f819eb7/realization-99/.fs_version ls: cannot access storage/ensembles/d1ec9d3d-dba6-4a8d-b2fc-931f4f819eb7/realization-99/.fs_version: No such file or directory kenu@st-linrgs099 ~/drogon-dev/ert/output/01_drogon_ahm %

[KetilN]

THANKS ALOT ! I see the version file now:

ert/output/01_drogon_ahm/storage/.fs_version

kenu@st-linrgs146 ~/drogon-dev % more ert/output/01_drogon_ahm/storage/.fs_version {"version": 0} kenu@st-linrgs146 ~/drogon-dev %

Then I guess you may close this GITHUB issue.

I can create a new issue (and refer to this if relevant) if I see similar error message popping up.

[KetilN]

Just for the record: Right now: Still same "Request error" in "komodo 2023.03.02", when plotting a "data type like NPV@0" without data in the default_X cases (cases from AHM) when the storage also contains the data from the pred_ref case, ref:

[andreas-el]

I've already mentioned to @oyvindeide (Sloths) that the .fs_version file is not found in bleeding, but present in earlier versions. This is a subject that should and will be further developed in the future as far as i understand.

I think there is a major difference between the march-releases and bleeding with respect to storage. This is likely the reason you see differences between the komodo-versions. Thanks for following up on this.