Closed oloapinivad closed 5 years ago
Same issue here with zg_6hrPlevPt and wap_6hrPlev using tables from https://github.com/PCMDI/cmip6-cmor-tables.git at commit 0f8a468d5929d1130a48c1ef28cf1b94700fa888.
Sorry if I come back on this but this is quite important: as long as I know passing PrePARE is a mandatory requirement for ESGF data publication.
These two variables wap_6hrPlev
and zg_6hrPlevPt
(present in the AOGCM historical) are still reporting the following errors:
The entry wap could not be found in CMOR table
The entry zg could not be found in CMOR table
Looking at the corresponding cmor tables suggest that it is merely a naming issue.
I think the variable name should be wap4
instead of wap
and zg27
instead of zg
.
@oloapinivad can you tell us which table - variable combinations you have trouble with PrePARE, and whether you expect some BUG at the ece2cmor
side or at the cmor table
side, if you have any idea?
This happening only for wap_6hrPlev
and zg_6hrPlevPt
It seems that the variable name is including the number of levels in the tables but not in the ece2cmor3 output. Frankly I cannot tell on which side is the problem.
If you look for example at https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_6hrPlevPt.json you will see that zg
is not there, but only zg27
, zg7h
and zg500
. I imagine that since multiple variables pointing to zg
exist in such table they were obliged to use naming which includes the number of levels: this is not currently handled by ece2cmor3 as long as I know.
This is a bug in PrePARE: the variables zg27
, zg7h
and zg500
all have the same out_name: zg
, which is the name that appears in our file name and netcdf file. This means (i) we should only take one of them during cmorization since otherwise they overwrite eachother (this has been taken care of in the refactoring of the task loader) and (ii) PrePARE should check on the out_name
field, not the actual variable name. This is however difficult for PrePARE because it wants to know the exact metadata of the variable for checking it. So we should address this with the PCMDI CMOR-people IMO.
I see what you mean. What I am not understanding completely is why when we opened such issue here https://github.com/PCMDI/cmor/issues/409 they said it was fixed why it seems not the case. I will reopen the issue.
I've just seen this for a file for ta7h in E3hrPt, which is named ta_E3hrPt_CNRM-CM6-1_highres-future_r1i1p1f2_gr_201501010300-201701010000.nc. The file passes in PrePARE from CMOR 3.3.2 but fails with CMOR 3.4.0.
Just to let you know that I still have no feedback from PCMDI https://github.com/PCMDI/cmor/issues/502
@oloapinivad which cmor version are you using?
More precisely: in ~/.qa-dkrz/config.txt
you will find a line like
PrePARE=<base_path>/bin/PrePARE
Then you should be able to check in
<base_path>/lib/python2.7/site-packages/cmip6_cv/PrePARE/out_names_tests.json
if that file contains the missing entries for 6hrPlev_wap
and 6hrPlevPt_zg
.
They indeed have been added to the original file in cmor some 8 months ago, but apparently it takes some time to trickle down into the regular installation. Since we are waiting on cmor 3.5.0 for #421 it should come at some point.
For now there are three options
Wow, well spotted, great!
I am using cmor 3.4.0 via conda and as expected there is no entry for 6hrPlev_wap
and 6hrPlevPt_zg
.
Personally, I would update manually the json file. When 3.5.0 will be released we can forget this limitation...
Go ahead, I think that is a good way to address the issue while waiting for cmor 3.5.0.
Unfortunately there has been also other changes in the code that should be backported to 3.4.0
The code is now crashing with:
Checking data, please wait...
'checkCMIP6' object has no attribute 'has_4_pressure_levels'
^[[1;34;47m└──> :: SKIPPED :: /scratch/ms/it/ccpd/ece3/chis/cmorized/cmor_1853/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1/6hrPlev/wap/gr/v20190626/wap_6hrPlev_EC-Earth3_historical_r4i1p1f1_gr_185301010300-185312312100.nc^[[0m
Yes, sorry, should have seen that coming. Did the run continue? If so the 6hrPlevPt_zg
part should have worked, confirming our theory.
So it is probably the heat wave but I am quite confused. I am working directly with PrePARE right now but the same can be obtained of course with QA-DKRZ
6hrPlev_wap
passes PrePARE when I backport both the new function in PrePARE.py (the new function has_4_pressure_levels
here) and the correct out_names_tests.json
However, even with such changes, the 6hrPlevPt_zg
still crashes The entry zg could not be found in CMOR table
which is particularly weird because it is there in the json file.
Hm. Good that you got wap working! As for zg, that is a bit strange. One difference I see is that there are different zg
s in the same table. I don't quite get the rationale for that. Need to do some more digging. Could you try the nightly or github version of cmor in the meantime?
I found out setting up the nightly built was quite easy. However, problems are not solved so I guess I am doing something wrong. I create a new cmor-nightly environment from here https://cmor.llnl.gov/mydoc_cmor3_conda/
conda create -n cmor-nightly -c pcmdi/label/nightly -c conda-forge cmor
The current list of packages seems to point to a very recent version of cmor (21/06/19):
conda list -n cmor-nightly
# packages in environment at /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor-nightly:
#
# Name Version Build Channel
bzip2 1.0.6 h14c3975_1002 conda-forge
ca-certificates 2019.6.16 hecc5488_0 conda-forge
certifi 2019.6.16 py27_0 conda-forge
cftime 1.0.3.4 py27hd352d35_1001 conda-forge
cmor 2019.06.21.master.numpy py27h9ac9557_0 pcmdi/label/nightly
curl 7.64.1 hf8cf82a_0 conda-forge
expat 2.2.5 he1b5a44_1003 conda-forge
hdf4 4.2.13 h9a582f1_1002 conda-forge
hdf5 1.10.5 nompi_h3c11f04_1100 conda-forge
jpeg 9c h14c3975_1001 conda-forge
json-c 0.13.1 h14c3975_1001 conda-forge
krb5 1.16.3 h05b26f9_1001 conda-forge
libblas 3.8.0 7_openblas conda-forge
libcblas 3.8.0 7_openblas conda-forge
libcurl 7.64.1 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
liblapack 3.8.0 7_openblas conda-forge
libnetcdf 4.6.2 h056eaf5_1002 conda-forge
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.1.0 hdf63c60_0
libuuid 2.32.1 h14c3975_1000 conda-forge
ncurses 6.1 hf484d3e_1002 conda-forge
netcdf4 1.5.1.2 py27h73a1b54_1 conda-forge
numpy 1.16.4 py27h95a1406_0 conda-forge
openblas 0.3.5 h9ac9557_1001 conda-forge
openssl 1.1.1b h14c3975_1 conda-forge
pip 19.1.1 py27_0 conda-forge
python 2.7.15 h721da81_1008 conda-forge
readline 7.0 hf8c457e_1001 conda-forge
setuptools 41.0.1 py27_0 conda-forge
six 1.12.0 py27_1000 conda-forge
sqlite 3.28.0 h8b20d00_0 conda-forge
tk 8.6.9 hed695b0_1002 conda-forge
udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge
wheel 0.33.4 py27_0 conda-forge
zlib 1.2.11 h14c3975_1004 conda-forge
However, we still get the same error!!!!
ccpd@cca-login4:~> source activate cmor-nightly
(cmor-nightly) ccpd@cca-login4:~> which PrePARE
/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor-nightly/bin/PrePARE
(cmor-nightly) ccpd@cca-login4:~> PrePARE --table-path ~/perm/ecearth3/cmorization/ece2cmor3/ece2cmor3/resources/tables ~/scratch/ece3/chis/cmorized/cmor_1850/
=====================================================================================
The entry zg could not be found in CMOR table
=====================================================================================
=====================================================================================
The entry wap could not be found in CMOR table
=====================================================================================
Number of files scanned: 264
Number of file with error(s): 2
Of course the PrePARE.py
and the out_names_tests.json
are correctly updated, so I am really wondering what is going on there... perhaps a mixing between conda packages?
So after a bit of digging thanks to PCDMI people we manage to solve this. It was a bug in PrePARE as mentioned here (https://github.com/PCMDI/cmor/issues/502#issuecomment-507373081) that now is fixed and that will be in nightly soon (and then in cmor 3.5.0).
Most importantly, it is not a problem in our data. I am closing this, feel free to reopen if you think it is the case
Do I get it right that this only requires an update to cmor 3.5.0
for PrePARE?
Or is the problem only solved if at the same time ece2cmor
is updated for cmor 3.5.0
? In the later case this issue needs to be reopened and to be labeled with cmor 3.5.0
.
As long as I understand it is simply a bug in PrePARE.py
and out_names_tests.json
from the cmor package. Updating these files fixes problem. Current output from ece2cmor3
is fine.
Great work, @oloapinivad! Just to second that: Yes, this was only a problem for QA, not for ece2cmor.
Hi all,
I am running v1.0.0 with what I think the most updated cmor tables. I produce 265 variables from the historical cmip6 for AOGCM varlist.
The only validation tool that I am aware of so far is PrePARE, and this is failing for wap_6hrPlev and zg_6hrPlevPt stating that variables are not in the table.
Indeed, zg is not there, but we have zg27, zg7h or zg500. Same for wap that is wap4. This seems to be the same issue we had for PRIMAVERA in #275.
Does anyone still use PrePARE and had the same issue? Thanks Paolo