search

ecmwf / ecpoint-calibrate

Interactive GUI (developed in Python) for calibration and conditional verification of numerical weather prediction model outputs.
GNU General Public License v3.0
21 stars 8 forks source link

some issues #124

Closed AugustinVintzileos closed 3 years ago

AugustinVintzileos commented 3 years ago

Due to some issues I had with version 12 Esti and I decided to deleted the previously downloaded extended range reforecast and start fresh. In fact it took some babysitting deleting the files as:

mova@naglfar:/vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC> \rm -r * rm: remove write-protected regular file 'cdir/._mviconlist'? y rm: remove write-protected regular file 'cdir/2019010100/._mviconlist'? y rm: remove write-protected regular file 'cdir/2019010100/._mvviewinfo'? y rm: remove write-protected regular file 'cdir/._mvviewinfo'? y rm: remove write-protected regular file 'sd/._mviconlist'? y

Initially I was happy thinking that that these .file files could interact somehow with the software. Re-downloading the reforecast from MARS took less than one day.

So I ran ecPoint v12:

mova@naglfar:~/ecPoint> ./ecPoint-Calibrate-0.12.0.AppImage Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank" Detected volume bindings: /home/mo/mova:/home,/media:/media,/vol:/vol,/tmp:/tmp,/var/tmp:/var/tmp,/scratch:/scratch Stopping container: bed8a45141f5 Stopping container: 7e94a570c359 Pulling from onyb/ecpoint-calibrate-logger Digest: sha256:8138775401749bde6d8730e15e132067976612c587fd5960e792af799fb7cee9 Status: Image is up to date for onyb/ecpoint-calibrate-logger:0.12.0 Pulling from onyb/ecpoint-calibrate-core Digest: sha256:55efa2808580a3c97e08612b6d6e6fb93820465f2642a6e5cb691f7efde147e3 Status: Image is up to date for onyb/ecpoint-calibrate-core:0.12.0 Running Docker container: image=onyb/ecpoint-calibrate-logger:0.12.0 containerID=238d7d97e805 Running Docker container: image=onyb/ecpoint-calibrate-core:0.12.0 containerID=13f640bec92d

Then I started generating a new table. The code reads the OBS ok but when I am trying to to read the predictant (2t) I get a red warning "500 internal server error" and the following messages:

Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged. Exception on /get-predictor-units [POST] Traceback (most recent call last): File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/app/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function return cors_after_request(app.make_response(f(*args, kwargs))) File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/app/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/app/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](req.view_args) File "/app/core/api.py", line 256, in get_predictor_units units = get_units(path) File "/app/core/api.py", line 343, in get_units first_grib_file = next(base_predictor_path.glob("*/.grib")) StopIteration 172.17.0.1 - - [06/Nov/2020 22:50:40] "POST /get-predictor-units HTTP/1.1" 500 -

Same for reading the predictors. Then I go to the define predictors part. Impossible to define any predictor e.g., BLH though everything has the correct access modes. For example:

mova@naglfar:/vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC> ls -lt total 629 drwxr-x--- 732 mova mo 732 Nov 6 20:58 swvl4 drwxr-x--- 732 mova mo 732 Nov 6 20:57 swvl3 drwxr-x--- 732 mova mo 732 Nov 6 20:56 swvl2 drwxr-x--- 732 mova mo 732 Nov 6 20:55 swvl1 drwxr-x--- 732 mova mo 732 Nov 6 20:53 skt drwxr-x--- 732 mova mo 732 Nov 6 20:52 sd drwxr-x--- 732 mova mo 732 Nov 6 20:50 lsm drwxr-x--- 732 mova mo 732 Nov 6 20:49 lcc drwxr-x--- 732 mova mo 732 Nov 6 20:47 fal drwxr-x--- 732 mova mo 732 Nov 6 20:46 cdir drwxr-x--- 732 mova mo 732 Nov 6 20:44 cape drwxr-x--- 732 mova mo 732 Nov 6 20:42 blh drwxr-x--- 732 mova mo 732 Nov 6 20:41 2t drwxr-x--- 732 mova mo 732 Nov 6 20:39 10v drwxr-x--- 732 mova mo 732 Nov 6 20:37 10u drwxr-x--- 732 mova mo 732 Nov 6 20:36 100v drwxr-x--- 732 mova mo 732 Nov 6 20:35 100u

and so on until the last subdirectory

it looks like the error output line: "Exception on /get-predictor-units [POST]" is critical. Somehow we cannot read the units from the grib files?

FatimaPillosu commented 3 years ago

Hi @AugustinVintzileos , Can I ask you to let me know where is the data that you are using, so I can try to create the table myself and see whether I can reproduce the error. I'll do this over the weekend. Cheers, Fatima

AugustinVintzileos commented 3 years ago

Hi Fatima,

/vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC

It looks like there is group access but please let me now if you have any issues seeing or reading the data, I'll be monitoring my email during the weekend.

Augustin

From: Fatima Pillosu notifications@github.com Sent: Friday, November 6, 2020 6:14 PM To: esowc/ecPoint-Calibrate ecPoint-Calibrate@noreply.github.com Cc: Augustin Vintzileos Augustin.Vintzileos@ecmwf.int; Mention mention@noreply.github.com Subject: Re: [esowc/ecPoint-Calibrate] some issues (#124)

Hi @AugustinVintzileoshttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAugustinVintzileos&data=04%7C01%7CAugustin.Vintzileos%40ecmwf.int%7Cb16198735b5849eca85308d882a9b6a5%7C21b711c6aab74d369ffbac0357bc20ba%7C0%7C0%7C637403012705464833%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VFkmtbTfPnqz6p5aigdAraw28NKI%2F6WWEnPODwHuvi8%3D&reserved=0 , Can I ask you to let me know where is the data that you are using, so I can try to create the table myself and see whether I can reproduce the error. I'll do this over the weekend. Cheers, Fatima

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fesowc%2FecPoint-Calibrate%2Fissues%2F124%23issuecomment-723341227&data=04%7C01%7CAugustin.Vintzileos%40ecmwf.int%7Cb16198735b5849eca85308d882a9b6a5%7C21b711c6aab74d369ffbac0357bc20ba%7C0%7C0%7C637403012705474789%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FYIZcd84SXUEhEj3B%2Fon5QR7FgAzsIbnSKzMz0q0Gc0%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARHK63MIWXG6AUXTTSNXVF3SOR7NHANCNFSM4TNHMY3A&data=04%7C01%7CAugustin.Vintzileos%40ecmwf.int%7Cb16198735b5849eca85308d882a9b6a5%7C21b711c6aab74d369ffbac0357bc20ba%7C0%7C0%7C637403012705474789%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=G%2FiA4YMSrpdJ7zO8uD%2BkWcNG%2BHBOReSOh2Gz2yDIoWc%3D&reserved=0.

onyb commented 3 years ago

What does the directory structure look like, under /vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC?

My dataset is arranged like this:

➜  ecPoint-Calibrate git:(master) tree /Volumes/Pearl/ECMWF/ESoWC/FC/ 
/Volumes/Pearl/ECMWF/ESoWC/FC/
├── cape
│   ├── 2015060100
│   │   ├── cape_20150601_00_03.grib
│   │   ├── cape_20150601_00_04.grib
│   │   ├── cape_20150601_00_05.grib
│   │   ├── cape_20150601_00_06.grib

Can you please run the following command and upload the resulting out.txt file?

tree /vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC -o out.txt
AugustinVintzileos commented 3 years ago

Hi Anirudha,

The directory structure bellow FC looks the same with yours e.g., FC/cape/2015060100/cape_20150601_00_03.grib

Is there some module that I must load for running the command tree ?

Augustin

From: Anirudha Bose notifications@github.com Sent: Sunday, November 8, 2020 7:17 AM To: esowc/ecPoint-Calibrate ecPoint-Calibrate@noreply.github.com Cc: Augustin Vintzileos Augustin.Vintzileos@ecmwf.int; Mention mention@noreply.github.com Subject: Re: [esowc/ecPoint-Calibrate] some issues (#124)

What does the directory structure look like, under /vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC?

My dataset is arranged like this:

➜ ecPoint-Calibrate git:(master) tree /Volumes/Pearl/ECMWF/ESoWC/FC/

/Volumes/Pearl/ECMWF/ESoWC/FC/

├── cape

│ ├── 2015060100

│ │ ├── cape_20150601_00_03.grib

│ │ ├── cape_20150601_00_04.grib

│ │ ├── cape_20150601_00_05.grib

│ │ ├── cape_20150601_00_06.grib

Can you please run the following command and upload the resulting out.txt file?

tree /vol/highlander/ecPoint_Calibrate_InDB/ecPoint_Temperature/ECMWF_47r1_extrange/FC -o out.txt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fesowc%2FecPoint-Calibrate%2Fissues%2F124%23issuecomment-723569621&data=04%7C01%7CAugustin.Vintzileos%40ecmwf.int%7Cd76207411ee846e3e4fd08d883e0402c%7C21b711c6aab74d369ffbac0357bc20ba%7C0%7C0%7C637404346452518325%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yxVX25oT36VcHVNsNyYMsww1pzEqTRsa9H9Zk87IT0Y%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARHK63P2MAFZPTCOM6OIXRLSO2D5HANCNFSM4TNHMY3A&data=04%7C01%7CAugustin.Vintzileos%40ecmwf.int%7Cd76207411ee846e3e4fd08d883e0402c%7C21b711c6aab74d369ffbac0357bc20ba%7C0%7C0%7C637404346452528290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=soKHe23%2BnJzKo%2BycNPpemU3lT6hQMpnHo%2FUfwzhY6xY%3D&reserved=0.

FatimaPillosu commented 3 years ago

Hi all, we cannot run the command "tree". I don't know why. I get the same error of Augustin.

omicron{/home/mo/mofp/Downloads}:7 --> ./ecPoint-Calibrate-0.12.0.AppImage Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank" Detected volume bindings: /home/mo/mofp:/home,/media:/media,/vol:/vol,/tmp:/tmp,/var/tmp:/var/tmp,/scratch:/scratch Pulling from onyb/ecpoint-calibrate-core Digest: sha256:55efa2808580a3c97e08612b6d6e6fb93820465f2642a6e5cb691f7efde147e3 Status: Image is up to date for onyb/ecpoint-calibrate-core:0.12.0 Pulling from onyb/ecpoint-calibrate-logger Digest: sha256:8138775401749bde6d8730e15e132067976612c587fd5960e792af799fb7cee9 Status: Image is up to date for onyb/ecpoint-calibrate-logger:0.12.0 Running Docker container: image=onyb/ecpoint-calibrate-core:0.12.0 containerID=be3bc66e7064 Running Docker container: image=onyb/ecpoint-calibrate-logger:0.12.0 containerID=ce0415c8a8e9

Just to point out, it is not related to the fact that we are reading temperature as this doesn't happen with the temperature from ERA5. I tried again, just to double check.

So, I don't know where could be the problem.

I'm sending to @onyb a couple of days the extended range and ERA5 forecasts in the new google drive space that I created, so he can compare and analyze where might be the problem. I'm a bit lost at the moment.

Cheers,

Fatima

onyb commented 3 years ago

Is this issue resolved by fixing file permissions?

FatimaPillosu commented 3 years ago

yes!