wap_6hrPlev & zg_6hrPlevPt not passing PrePARE

oloapinivad commented 5 years ago

Hi all,

I am running v1.0.0 with what I think the most updated cmor tables. I produce 265 variables from the historical cmip6 for AOGCM varlist.

The only validation tool that I am aware of so far is PrePARE, and this is failing for wap_6hrPlev and zg_6hrPlevPt stating that variables are not in the table.

Indeed, zg is not there, but we have zg27, zg7h or zg500. Same for wap that is wap4. This seems to be the same issue we had for PRIMAVERA in #275.

Does anyone still use PrePARE and had the same issue? Thanks Paolo

jmrgonza commented 5 years ago

Same issue here with zg_6hrPlevPt and wap_6hrPlev using tables from https://github.com/PCMDI/cmip6-cmor-tables.git at commit 0f8a468d5929d1130a48c1ef28cf1b94700fa888.

oloapinivad commented 5 years ago

Sorry if I come back on this but this is quite important: as long as I know passing PrePARE is a mandatory requirement for ESGF data publication. These two variables wap_6hrPlev and zg_6hrPlevPt (present in the AOGCM historical) are still reporting the following errors:

The entry wap could not be found in CMOR table
The entry zg could not be found in CMOR table

Looking at the corresponding cmor tables suggest that it is merely a naming issue. I think the variable name should be wap4 instead of wap and zg27 instead of zg.

treerink commented 5 years ago

@oloapinivad can you tell us which table - variable combinations you have trouble with PrePARE, and whether you expect some BUG at the ece2cmor side or at the cmor table side, if you have any idea?

oloapinivad commented 5 years ago

This happening only for wap_6hrPlev and zg_6hrPlevPt

It seems that the variable name is including the number of levels in the tables but not in the ece2cmor3 output. Frankly I cannot tell on which side is the problem.

If you look for example at https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_6hrPlevPt.json you will see that zg is not there, but only zg27, zg7h and zg500. I imagine that since multiple variables pointing to zg exist in such table they were obliged to use naming which includes the number of levels: this is not currently handled by ece2cmor3 as long as I know.

goord commented 5 years ago

This is a bug in PrePARE: the variables zg27, zg7h and zg500 all have the same out_name: zg, which is the name that appears in our file name and netcdf file. This means (i) we should only take one of them during cmorization since otherwise they overwrite eachother (this has been taken care of in the refactoring of the task loader) and (ii) PrePARE should check on the out_name field, not the actual variable name. This is however difficult for PrePARE because it wants to know the exact metadata of the variable for checking it. So we should address this with the PCMDI CMOR-people IMO.

oloapinivad commented 5 years ago

I see what you mean. What I am not understanding completely is why when we opened such issue here https://github.com/PCMDI/cmor/issues/409 they said it was fixed why it seems not the case. I will reopen the issue.

jonseddon commented 5 years ago

I've just seen this for a file for ta7h in E3hrPt, which is named ta_E3hrPt_CNRM-CM6-1_highres-future_r1i1p1f2_gr_201501010300-201701010000.nc. The file passes in PrePARE from CMOR 3.3.2 but fails with CMOR 3.4.0.

oloapinivad commented 5 years ago

Just to let you know that I still have no feedback from PCMDI https://github.com/PCMDI/cmor/issues/502

zklaus commented 5 years ago

@oloapinivad which cmor version are you using? More precisely: in ~/.qa-dkrz/config.txt you will find a line like

PrePARE=<base_path>/bin/PrePARE

Then you should be able to check in

<base_path>/lib/python2.7/site-packages/cmip6_cv/PrePARE/out_names_tests.json

if that file contains the missing entries for 6hrPlev_wap and 6hrPlevPt_zg. They indeed have been added to the original file in cmor some 8 months ago, but apparently it takes some time to trickle down into the regular installation. Since we are waiting on cmor 3.5.0 for #421 it should come at some point.

For now there are three options

ignore the tag in qa-dkrz
add the entries manually
move to a github version of cmor (for qa-dkrz)

oloapinivad commented 5 years ago

Wow, well spotted, great!

I am using cmor 3.4.0 via conda and as expected there is no entry for 6hrPlev_wap and 6hrPlevPt_zg. Personally, I would update manually the json file. When 3.5.0 will be released we can forget this limitation...

zklaus commented 5 years ago

Go ahead, I think that is a good way to address the issue while waiting for cmor 3.5.0.

oloapinivad commented 5 years ago

Unfortunately there has been also other changes in the code that should be backported to 3.4.0

The code is now crashing with:

Checking data, please wait...
'checkCMIP6' object has no attribute 'has_4_pressure_levels'
^[[1;34;47m└──> :: SKIPPED    :: /scratch/ms/it/ccpd/ece3/chis/cmorized/cmor_1853/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1/6hrPlev/wap/gr/v20190626/wap_6hrPlev_EC-Earth3_historical_r4i1p1f1_gr_185301010300-185312312100.nc^[[0m

zklaus commented 5 years ago

Yes, sorry, should have seen that coming. Did the run continue? If so the 6hrPlevPt_zg part should have worked, confirming our theory.

oloapinivad commented 5 years ago

So it is probably the heat wave but I am quite confused. I am working directly with PrePARE right now but the same can be obtained of course with QA-DKRZ

6hrPlev_wap passes PrePARE when I backport both the new function in PrePARE.py (the new function has_4_pressure_levels here) and the correct out_names_tests.json
However, even with such changes, the 6hrPlevPt_zg still crashes The entry zg could not be found in CMOR table which is particularly weird because it is there in the json file.

zklaus commented 5 years ago

Hm. Good that you got wap working! As for zg, that is a bit strange. One difference I see is that there are different zgs in the same table. I don't quite get the rationale for that. Need to do some more digging. Could you try the nightly or github version of cmor in the meantime?

oloapinivad commented 5 years ago

I found out setting up the nightly built was quite easy. However, problems are not solved so I guess I am doing something wrong. I create a new cmor-nightly environment from here https://cmor.llnl.gov/mydoc_cmor3_conda/

conda create -n cmor-nightly -c pcmdi/label/nightly -c conda-forge cmor

The current list of packages seems to point to a very recent version of cmor (21/06/19):

conda list -n cmor-nightly
# packages in environment at /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor-nightly:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.6             h14c3975_1002    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py27_0    conda-forge
cftime                    1.0.3.4         py27hd352d35_1001    conda-forge
cmor                      2019.06.21.master.numpy  py27h9ac9557_0    pcmdi/label/nightly
curl                      7.64.1               hf8cf82a_0    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
libblas                   3.8.0                7_openblas    conda-forge
libcblas                  3.8.0                7_openblas    conda-forge
libcurl                   7.64.1               hda55be3_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0                7_openblas    conda-forge
libnetcdf                 4.6.2             h056eaf5_1002    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libuuid                   2.32.1            h14c3975_1000    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
netcdf4                   1.5.1.2          py27h73a1b54_1    conda-forge
numpy                     1.16.4           py27h95a1406_0    conda-forge
openblas                  0.3.5             h9ac9557_1001    conda-forge
openssl                   1.1.1b               h14c3975_1    conda-forge
pip                       19.1.1                   py27_0    conda-forge
python                    2.7.15            h721da81_1008    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
setuptools                41.0.1                   py27_0    conda-forge
six                       1.12.0                py27_1000    conda-forge
sqlite                    3.28.0               h8b20d00_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
udunits2                  2.2.27.6          h4e0c4b3_1001    conda-forge
wheel                     0.33.4                   py27_0    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

However, we still get the same error!!!!

ccpd@cca-login4:~>  source activate cmor-nightly
(cmor-nightly) ccpd@cca-login4:~> which PrePARE
/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor-nightly/bin/PrePARE
(cmor-nightly) ccpd@cca-login4:~> PrePARE --table-path ~/perm/ecearth3/cmorization/ece2cmor3/ece2cmor3/resources/tables ~/scratch/ece3/chis/cmorized/cmor_1850/

=====================================================================================
The entry zg could not be found in CMOR table
=====================================================================================

=====================================================================================
The entry wap could not be found in CMOR table
=====================================================================================

Number of files scanned: 264
Number of file with error(s): 2

Of course the PrePARE.py and the out_names_tests.json are correctly updated, so I am really wondering what is going on there... perhaps a mixing between conda packages?

oloapinivad commented 5 years ago

So after a bit of digging thanks to PCDMI people we manage to solve this. It was a bug in PrePARE as mentioned here (https://github.com/PCMDI/cmor/issues/502#issuecomment-507373081) that now is fixed and that will be in nightly soon (and then in cmor 3.5.0).

Most importantly, it is not a problem in our data. I am closing this, feel free to reopen if you think it is the case

treerink commented 5 years ago

Do I get it right that this only requires an update to cmor 3.5.0 for PrePARE? Or is the problem only solved if at the same time ece2cmor is updated for cmor 3.5.0? In the later case this issue needs to be reopened and to be labeled with cmor 3.5.0.

oloapinivad commented 5 years ago

As long as I understand it is simply a bug in PrePARE.py and out_names_tests.json from the cmor package. Updating these files fixes problem. Current output from ece2cmor3 is fine.

zklaus commented 5 years ago

Great work, @oloapinivad! Just to second that: Yes, this was only a problem for QA, not for ece2cmor.

EC-Earth / ece2cmor3

wap_6hrPlev & zg_6hrPlevPt not passing PrePARE #462