search

IS-ENES-Data / QA-DKRZ

Quality and CF checker of meta-data in climate related data sets (NetCDF files)
4 stars 4 forks source link

CV test failing #11

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

I am running QA-DKRZ on CMIP6 data. The CV test is failing with the following error:

(qa-dkrz) ccpd@cca-login4:~> qa-dkrz -P CMIP6 -f QA_conf.conf -m
NEXT: rsdt_Amon_EC-Earth3_historical_r4i1p1f1_gr_185001-185012.ncterminate called after throwing an instance of 'Json::LogicError'
  what():  in Json::Value::operator[](ArrayIndex)const: requires arrayValue

Traceback (most recent call last):
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa-dkrz.py", line 676, in <module>
    run()
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa-dkrz.py", line 568, in run
    if not launch_list[0].start(queue,):
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa_exec.py", line 728, in start
    if not self.run(t_vars):
  File "/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa_exec.py", line 583, in run
    period      = log_entry['period'],

Current configuration read as:

(qa-dkrz) ccpd@cca-login4:~> qa-dkrz -P CMIP6 -f QA_conf.conf -e_show_conf
APPLY_MAXIMUM_DATE_RANGE=t
ARITHMETIC_MEAN=t
CF=1.7
CFG_FILE=/home/ms/it/ccpd/.qa-dkrz/config.txt
CF_AREA_TYPES=cf-area-type-table.txt
CF_CHECK_LIST=CF_check-list.conf
CF_FOLLOW_RECOMMENDATIONS=t
CF_STANDARD_NAMES=cf-standard-name-table.xml
CF_STD_NAME_VERSION=std-name=v66,15May.2019,region=v4,18Dec.2018,area=v9,07Aug.2018
CF_STD_REGION_NAMES=cf-standardized-region-list.txt
CF_VERSION=CF_STD_NAME_VERSION:std-name=v66,15May.2019,region=v4,18Dec.2018,area=v9,07Aug.2018
CHECK_MODE=DATA
CMIP6_VERSION=CMIP6_CVs:master-89827a6|CMIP6_MIP_tables.xlsx:2019-03-11|cmip6-cmor-tables:master-fec7f51
CONDA=True
CONDA_CMOR_ENV=/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor
CONDA_PATH=/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/bin/conda
CONDA_QA_ENV=/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz
CORDEX_VERSION=Missing
CT_PATH_INDEX=[2, 3, 4]
CURR_DIR=/home/ms/it/ccpd
DEFAULT_VALID_MAX=1E+17
DEFAULT_VALID_MIN=-1E-15
DRS_PATH_BASE=CMIP6
EXCLUDE_ATTRIBUTE=['comment', 'contact', 'history', 'references', 'title']
FILE_NAME_SEP=_
FILE_NAME_VAR_INDEX=0
FREQ_INDEX_PATH=5
HARD_SLEEP_PERIOD=10
IGNORE_REFERENCE_DATE=t
IGNORE_REF_DATE_ACROSS_EXP=t
LOCK_PATH_LIST=[]
LOCK_VAR_LIST=[]
LOG_FILE_INDEX=[2, 1, 3]
LOG_PATH_INDEX=[2, 3, 4, 5]
MAIL=mailx
NEVER_BREAK_SESSION=t
NEXT_CF_UPDATE=20190517+1
NEXT_CMIP6_UPDATE=20190517+1
NUM_EXEC_THREADS=1
PATH_SEP=/
PROJECT=CMIP6
PROJECT_DATA=/lus/snx11062/scratch/ms/it/ccpd/ece3/ch00/cmorized/Year_1850
PROJECT_TABLE_PREFIX=pt
PrePARE=/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/ece2cmor/bin/PrePARE
QA_CHECK_LIST=CMIP6_check-list.conf
QA_CONF=QA_conf.conf,/home/ms/it/ccpd/.qa-dkrz/QA_TABLES/tables/projects/CMIP6/CMIP6_qa.conf
QA_EXEC_HOSTS=cca-login4
QA_HOST=cca-login4
QA_RESULTS=/lus/snx11062/scratch/ms/it/ccpd/tmp_ecearth3/QA
QA_SRC=/lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz
QA_TABLES=/home/ms/it/ccpd/.qa-dkrz/QA_TABLES
QA_VERSION=qa-dkrz-0.6.7-59,master-5fdfa30
QUERY_EMPTY_DIR=t
QUERY_EMPTY_FILE=t
QUERY_NON_NC_FILE=t
QUERY_ONLY_NC=t
REATTEMPT_LIMIT=5
RUN_PREPARE=t
SELECT_PATH_LIST=['CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1']
SELECT_VAR_LIST=['.*']
SHOW_CONF=True
SLEEP_PERIOD=300
TABLE_DRS_CV=CMIP6_DRS_CV.csv
TABLE_EXPERIMENT=CMIP6_experiment_table.txt
TABLE_TIME_SCHEDULE=CMIP6_time_table.csv
TABLE_VAR_REQ=CMIP6_MIP_tables
TASK=QA_conf.conf
XLSX2CSV=/home/ms/it/ccpd/.qa-dkrz/QA_TABLES/xlsx2csv/xlsx2csv.py

As long as I am able to test, all the other tests as TIME, CF, DATA and CNSTY are correctly working. Do you have any idea if there is anything which is uncorrectly set? Thanks

uwefladrich commented 5 years ago

I found this issue, which seems to trigger the same error message. But I haven't a clue if the context is similar ;-)

zklaus commented 5 years ago

@oloapinivad is this still valid?

oloapinivad commented 5 years ago

With the conda version qa-dkrz-0.6.7-59,master-5fdfa30 when I set CHECK_MODE=CV I still get 

NEXT: tos_Oday_EC-Earth3_historical_r4i1p1f1_gn_18500101-18501231.ncterminate called after throwing an instance of 'Json::LogicError'
  what():  in Json::Value::operator[](ArrayIndex)const: requires arrayValue

So in one word, yes ;-)

h-dh commented 5 years ago

I would like to close this issue, because I think that is was fixed in the past. If you think that it is still present, please open a new issue.

Note that the conda package is pending. Cheers, hdh