search

IS-ENES-Data / QA-DKRZ

Quality and CF checker of meta-data in climate related data sets (NetCDF files)
4 stars 4 forks source link

CMOR Error: Udunits #14

Closed oloapinivad closed 5 years ago

oloapinivad commented 5 years ago

I am finding plenty of these udunits errors pointing to different aspect of the file units. A few examples here below: 

   - event:
       annotation: CMOR Error: Udunits: Error reading units system
       impact: L1
       tag: 881f
    - event:
       annotation: CMOR Error: Udunits: creating dimlessnew base unit
       impact: L1
       tag: d4b5
    - event:
       annotation: CMOR Error: Udunits: Error mapping dimless 'eq' unit
       impact: L1
       tag: c1d8
    - event:
       annotation: CMOR Error: Udunits: creating dimless unit
       impact: L1
       tag: 896b
    - event:
       annotation: CMOR Error: Udunits: Error mapping dimless unit
       impact: L1
       tag: 13f8
    - event:
       annotation: CMOR Error: Udunits: Error creating percent unit
       impact: L1
       tag: 7fc9
    - event:
       annotation: CMOR Error: Udunits: Error mapping percent unit
       impact: L1
       tag: 408c
    - event:
       annotation: CMOR Error: Udunits: creating psuunit unit
       impact: L1
       tag: 7799
    - event:
       annotation: CMOR Error: Udunits: Error mapping psu unit
       impact: L1
       tag: 7e60
    - event:
       annotation: CMOR Error: Udunits: creating dimless unit
       impact: L1
       tag: 896b
    - event:

I installed QA-DKRZ with conda and it seems to work fine. Current configuration is:

# Log-file created by QA-DKRZ
configuration:
 command-line:  /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa-dkrz.py -P CMIP6 -E PROJECT_DATA=/lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/linkdata -E SELECT=CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1 -E CHECK_MODE=TIME,DATA,CNSTY,CF,DRS -E QA_RESULTS=/lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results -E NUM_EXEC_THREADS=12 -m
 options:
  APPLY_MAXIMUM_DATE_RANGE: t
  ARITHMETIC_MEAN: t
  CF: 1.7
  CFG_FILE: /home/ms/it/ccpd/.qa-dkrz/config.txt
  CF_AREA_TYPES: cf-area-type-table.txt
  CF_CHECK_LIST: CF_check-list.conf
  CF_FOLLOW_RECOMMENDATIONS: t
  CF_STANDARD_NAMES: cf-standard-name-table.xml
  CF_STD_NAME_VERSION: std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018
  CF_STD_REGION_NAMES: cf-standardized-region-list.txt
  CF_VERSION: CF_STD_NAME_VERSION:std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018
  CHECK_MODE: ['TIME', 'DATA', 'CNSTY', 'CF', 'DRS']
  CMIP6_VERSION: CMIP6_CVs:master-55544b4|CMIP6_MIP_tables.xlsx:2019-05-20|CMOR:cmor-3.4.0-py27h17cb6a4_2|cmip6-cmor-tables:master-bc7d1c2
  CONDA: True
  CONDA_CMOR_ENV: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor
  CONDA_PATH: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/bin/conda
  CONDA_QA_ENV: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz
  CT_PATH_INDEX: [2, 3, 4]
  CT_PATH_INDEX_MAX: 4
  CURR_DIR: /home/ms/it/ccpd/post/ece3-postproc/easy2cmor3
  DEFAULT_VALID_MAX: 1E+17
  DEFAULT_VALID_MIN: -1E-15
  DRS_PATH_BASE: CMIP6
  EXCLUDE_ATTRIBUTE: ['comment', 'contact', 'history', 'references', 'title']
  FILE_NAME_SEP: _
  FILE_NAME_VAR_INDEX: 0
  FREQ_INDEX_PATH: 5
  HARD_SLEEP_PERIOD: 10
  IGNORE_REFERENCE_DATE: t
  IGNORE_REF_DATE_ACROSS_EXP: t
  LOCK_PATH_LIST: []
  LOCK_VAR_LIST: []
  LOG_FILE_INDEX: [2, 1, 3]
  LOG_FNAME_DIR: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results/check_logs
  LOG_INDEX_MAX: 5
  LOG_PATH_INDEX: [2, 3, 4, 5]
  MAIL: mailx
  NEVER_BREAK_SESSION: t
  NEXT_CF_UPDATE: 20190624+1
  NEXT_CMIP6_UPDATE: 20190624+1
  NUM_EXEC_THREADS: 12
  PATH_SEP: /
  PROJECT: CMIP6
  PROJECT_DATA: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/linkdata
  PROJECT_TABLE_PREFIX: pt
  PrePARE: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor/bin/PrePARE
  QA_CHECK_LIST: CMIP6_check-list.conf
  QA_CONF: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES/tables/projects/CMIP6/CMIP6_qa.conf
  QA_EXEC_HOSTS: ccappn013
  QA_HOST: ccappn013
  QA_RESULTS: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results
  QA_SRC: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz
  QA_TABLES: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES
  QA_VERSION: qa-dkrz-0.6.7-59,master-5fdfa30|CF_STD_NAME_VERSION:std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018|CMIP6_CVs:master-55544b4|CMIP6_MIP_tables.xlsx:2019-05-20|CMOR:cmor-3.4.0-py27h17cb6a4_2|cmip6-cmor-tables:master-bc7d1c2
  QUERY_EMPTY_DIR: t
  QUERY_EMPTY_FILE: t
  QUERY_NON_NC_FILE: t
  QUERY_ONLY_NC: t
  REATTEMPT_LIMIT: 5
  RUN_PREPARE: t
  SELECT: CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1
  SELECT_PATH_LIST: []
  SELECT_VAR_LIST: []
  SESSION: 2019-06-25T09:21:18
  SESSION_LOGDIR: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results/session_logs/2019-06-25T09:21:18
  SLEEP_PERIOD: 300
  STATUS_LINE: True
  STATUS_LINE_SZ: 69
  TABLES:
   - CF_AREA_TYPES: cf-area-type-table.txt
   - CF_CHECK_LIST: CF_check-list.conf
   - CF_STANDARD_NAMES: cf-standard-name-table.xml
   - CF_STD_REGION_NAMES: cf-standardized-region-list.txt
   - PROJECT_TABLE_PREFIX: pt
   - QA_CHECK_LIST: CMIP6_check-list.conf
   - TABLE_DRS_CV: CMIP6_DRS_CV.csv
   - TABLE_EXPERIMENT: CMIP6_experiment_table.txt
   - TABLE_TIME_SCHEDULE: CMIP6_time_table.csv
   - TABLE_VAR_REQ: CMIP6_MIP_tables
  XLSX2CSV: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES/xlsx2csv/xlsx2csv.py
  ZOMBIE_LIMIT: 3600

What could be the problem? Thanks for any hint. Best, Paolo

h-dh commented 5 years ago

Dear Paolo,

I cannot say much about this issue. The token 'CMOR Error:' in the annotations indicate that this is related to the PrePARE checker from the cmor package. As you certainly did, the cmor package has to be installed separately by conda create -n cmor -c conda-forge cmor

Then, you would have to run either

QA-DKRZ/install --up --force CMIP6 or your-path/miniconda2/envs/bin/qa-dkrz --install --up --force CMIP6

depending on the sources. cheers, hdh

On 6/25/19 11:39 AM, Paolo Davini wrote:

I am finding plenty of these |udunits| errors pointing to different aspect of the file units. A few examples here below:

|- event: annotation: CMOR Error: Udunits: Error reading units system impact: L1 tag: 881f - event: annotation: CMOR Error: Udunits: creating dimlessnew base unit impact: L1 tag: d4b5 - event: annotation: CMOR Error: Udunits: Error mapping dimless 'eq' unit impact: L1 tag: c1d8 - event: annotation: CMOR Error: Udunits: creating dimless unit impact: L1 tag: 896b - event: annotation: CMOR Error: Udunits: Error mapping dimless unit impact: L1 tag: 13f8 - event: annotation: CMOR Error: Udunits: Error creating percent unit impact: L1 tag: 7fc9 - event: annotation: CMOR Error: Udunits: Error mapping percent unit impact: L1 tag: 408c - event: annotation: CMOR Error: Udunits: creating psuunit unit impact: L1 tag: 7799 - event: annotation: CMOR Error: Udunits: Error mapping psu unit impact: L1 tag: 7e60 - event: annotation: CMOR Error: Udunits: creating dimless unit impact: L1 tag: 896b - event: |

I installed QA-DKRZ with conda and it seems to work fine. Current configuration is:

|# Log-file created by QA-DKRZ configuration: command-line: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz/python/qa-dkrz/qa-dkrz.py -P CMIP6 -E PROJECT_DATA=/lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/linkdata -E SELECT=CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1 -E CHECK_MODE=TIME,DATA,CNSTY,CF,DRS -E QA_RESULTS=/lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results -E NUM_EXEC_THREADS=12 -m options: APPLY_MAXIMUM_DATE_RANGE: t ARITHMETIC_MEAN: t CF: 1.7 CFG_FILE: /home/ms/it/ccpd/.qa-dkrz/config.txt CF_AREA_TYPES: cf-area-type-table.txt CF_CHECK_LIST: CF_check-list.conf CF_FOLLOW_RECOMMENDATIONS: t CF_STANDARD_NAMES: cf-standard-name-table.xml CF_STD_NAME_VERSION: std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018 CF_STD_REGION_NAMES: cf-standardized-region-list.txt CF_VERSION: CF_STD_NAME_VERSION:std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018 CHECK_MODE: ['TIME', 'DATA', 'CNSTY', 'CF', 'DRS'] CMIP6_VERSION: CMIP6_CVs:master-55544b4|CMIP6_MIP_tables.xlsx:2019-05-20|CMOR:cmor-3.4.0-py27h17cb6a4_2|cmip6-cmor-tables:master-bc7d1c2 CONDA: True CONDA_CMOR_ENV: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor CONDA_PATH: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/bin/conda CONDA_QA_ENV: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz CT_PATH_INDEX: [2, 3, 4] CT_PATH_INDEX_MAX: 4 CURR_DIR: /home/ms/it/ccpd/post/ece3-postproc/easy2cmor3 DEFAULT_VALID_MAX: 1E+17 DEFAULT_VALID_MIN: -1E-15 DRS_PATH_BASE: CMIP6 EXCLUDE_ATTRIBUTE: ['comment', 'contact', 'history', 'references', 'title'] FILE_NAMESEP: FILE_NAME_VAR_INDEX: 0 FREQ_INDEX_PATH: 5 HARD_SLEEP_PERIOD: 10 IGNORE_REFERENCE_DATE: t IGNORE_REF_DATE_ACROSS_EXP: t LOCK_PATH_LIST: [] LOCK_VAR_LIST: [] LOG_FILE_INDEX: [2, 1, 3] LOG_FNAME_DIR: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results/check_logs LOG_INDEX_MAX: 5 LOG_PATH_INDEX: [2, 3, 4, 5] MAIL: mailx NEVER_BREAK_SESSION: t NEXT_CF_UPDATE: 20190624+1 NEXT_CMIP6_UPDATE: 20190624+1 NUM_EXEC_THREADS: 12 PATH_SEP: / PROJECT: CMIP6 PROJECT_DATA: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/linkdata PROJECT_TABLE_PREFIX: pt PrePARE: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/cmor/bin/PrePARE QA_CHECK_LIST: CMIP6_check-list.conf QA_CONF: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES/tables/projects/CMIP6/CMIP6_qa.conf QA_EXEC_HOSTS: ccappn013 QA_HOST: ccappn013 QA_RESULTS: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results QA_SRC: /lus/snx11062/scratch/ms/it/ccpd/PRIMAVERA/anaconda2/envs/qa-dkrz/opt/qa-dkrz QA_TABLES: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES QA_VERSION: qa-dkrz-0.6.7-59,master-5fdfa30|CF_STD_NAME_VERSION:std-name=v67,18Jun.2019,region=v4,18Dec.2018,area=v9,07Aug.2018|CMIP6_CVs:master-55544b4|CMIP6_MIP_tables.xlsx:2019-05-20|CMOR:cmor-3.4.0-py27h17cb6a4_2|cmip6-cmor-tables:master-bc7d1c2 QUERY_EMPTY_DIR: t QUERY_EMPTY_FILE: t QUERY_NON_NC_FILE: t QUERY_ONLY_NC: t REATTEMPT_LIMIT: 5 RUN_PREPARE: t SELECT: CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r4i1p1f1 SELECT_PATH_LIST: [] SELECT_VAR_LIST: [] SESSION: 2019-06-25T09:21:18 SESSION_LOGDIR: /lus/snx11062/scratch/ms/it/ccpd/tmp_cmor/QA/chis/results/session_logs/2019-06-25T09:21:18 SLEEP_PERIOD: 300 STATUS_LINE: True STATUS_LINE_SZ: 69 TABLES: - CF_AREA_TYPES: cf-area-type-table.txt - CF_CHECK_LIST: CF_check-list.conf - CF_STANDARD_NAMES: cf-standard-name-table.xml - CF_STD_REGION_NAMES: cf-standardized-region-list.txt - PROJECT_TABLE_PREFIX: pt - QA_CHECK_LIST: CMIP6_check-list.conf - TABLE_DRS_CV: CMIP6_DRS_CV.csv - TABLE_EXPERIMENT: CMIP6_experiment_table.txt - TABLE_TIME_SCHEDULE: CMIP6_time_table.csv - TABLE_VAR_REQ: CMIP6_MIP_tables XLSX2CSV: /home/ms/it/ccpd/.qa-dkrz/QA_TABLES/xlsx2csv/xlsx2csv.py ZOMBIE_LIMIT: 3600 |

What could be the problem? Thanks for any hint. Best, Paolo

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/IS-ENES-Data/QA-DKRZ/issues/14?email_source=notifications&email_token=ACJZOW4F7QZGRFETEKB56V3P4HRUZA5CNFSM4H3GL7RKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G3P4T7Q, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJZOW5FS3EV4F2MHLI3WUDP4HRUZANCNFSM4H3GL7RA.

-- Dr. Heinz-Dieter Hollweg Abteilung Datenmanagement Deutsches Klimarechenzentrum GmbH (DKRZ) Bundesstraße 45a • D-20146 Hamburg • Germany

Phone: +49 40 460094-212 FAX: +49 40 460094-270 Email: hollweg@dkrz.de URL: www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig Sitz der Gesellschaft: Hamburg Amtsgericht Hamburg HRB 39784

oloapinivad commented 5 years ago

Thanks @h-dh. So apparently I had to specify the UDUNITS2_XML_PATH in my config.txt All the errors are gone and PrePARE is running correctlt.