search

PCMDI / cmor

Climate Model Output Rewriter
BSD 3-Clause "New" or "Revised" License
51 stars 33 forks source link

wap_6hrPlev and zg_6hrPlevPt failing PrePARE #502

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

Hi all,

sorry if I am opening a new issue on this but I am experiencing again the same problem I reported in #409 with PrePARE

Two variables wap_6hrPlev and zg_6hrPlevPt are still reporting the following errors when they are examined by PrePARE:

The entry wap could not be found in CMOR table
The entry zg could not be found in CMOR table

Looking at the corresponding cmor tables suggest that it is still the same kind of out_name issue (wap should be wap4, zg should zg27 at least in our case): could be possible that these two are still not included in the correct list?

thanks for any help! Best, Paolo

zklaus commented 5 years ago

I think we have found the origin of this and expect it is already fixed for 3.5.0, right @oloapinivad ?

mauzey1 commented 5 years ago

@oloapinivad This issue should have been solved by https://github.com/PCMDI/cmor/pull/412. The changes are currently in nightly and will be part of 3.5.0.

oloapinivad commented 5 years ago

Thanks @mauzey1. We successfully backported the wap_6hrPlev but we are still having problem with zg_6hrPlevPt. Anyway, I can close this.

zklaus commented 5 years ago

@oloapinivad, could you try with the current nightly or github version of cmor? If the issue persists there, we should keep this open and hope to get a fix into cmor 3.5.0.

oloapinivad commented 5 years ago

I am sorry but I need to reopen this issue. I installed the nightly build via conda (conda create -n cmor-nightly -c pcmdi/label/nightly -c conda-forge cmor) and run a test on the zg_6hrPlevPt variable:

ccpd@cca-login4:~> PrePARE --table-path ~/tables ~/scratch/ece3/chis/cmorized/cmor_1851/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1/6hrPlevPt/zg/gr/v20190626/zg_6hrPlevPt_EC-Earth3_historical_r4i1p1f1_gr_185101010000-185112311800.nc 

=====================================================================================
The entry zg could not be found in CMOR table
=====================================================================================
Number of files scanned: 1
Number of file with error(s): 1

However, if I force the variable name to zg27 it works fine: 

(cmor-nightly) ccpd@cca-login4:~> PrePARE --variable zg27 --table-path ~/tables ~/scratch/ece3/chis/cmorized/cmor_1851/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1/6hrPlevPt/zg/gr/v20190626/zg_6hrPlevPt_EC-Earth3_historical_r4i1p1f1_gr_185101010000-185112311800.nc 

Number of files scanned: 1
Number of file with error(s): 0

Did I do something wrong in the installation of the environment? Could anybody else confirm a similar issue or there is something wrong in my PrePARE? I can provide the file - or at least a portion of it - for some testing.

Thanks a lot for any hint.

mauzey1 commented 5 years ago

Yes, please provide a file that is having this problem.

oloapinivad commented 5 years ago

I subsampled a single day from the original file otherwise is too big for sharing it. I tested PrePARE from the cmor-nightly and I gott the same results. Here is the dropbox link

Let me know if I can help somehow. Thanks

mauzey1 commented 5 years ago

@oloapinivad @doutriaux1 Okay, I found the issue. When we replaced CDMS2 with NetCDF4 for reading files in PrePARE, we forgot to add a size attribute when accessing the pressure level dimension size. This should fix the issue.

https://github.com/PCMDI/cmor/blob/9a0d42106305ba9f5ae152ea90a5464c4901c2aa/LibCV/PrePARE/PrePARE.py#L310-L322

oloapinivad commented 5 years ago

Thanks @mauzey1 I tried to copy/paste your update into my PrePARE.py of my cmor-nightly conda environment but I still have the problem with zg. I guess this is working on the test file I sent you, so is there anything else different between my build and yours? Is this CDMS2 replacement more recent that the last nighty build? When a new nightly build will be released? Thanks

mauzey1 commented 5 years ago

@oloapinivad I found another issue. When there are multiple out_name checks for a variable in a table, PrePARE will loop through all of them. 6hrPlevPt_zg has two pressure level tests. If the first test passes then the second test will fail and set the variable name to the out_name. I changed it so that it will break out of the loop when it finds the passing test.

https://github.com/PCMDI/cmor/blob/e0498539f40f958752625f14ab5c3b39787d44e6/LibCV/PrePARE/PrePARE.py#L390-L403

oloapinivad commented 5 years ago

Great, now it is working! Thanks @mauzey1 I can definitely close this. A last question, when the next nightly build for conda will be available? The current one is from the 21/06/19