search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

CMCC-CM2: msftmz missing olevel and different name for basin (sector instead of basin) #1074

Open vincenzodetoma opened 3 years ago

vincenzodetoma commented 3 years ago

Describe the bug It's not possible to fix metadata for this case. I report the output, the recipe and the log file. Can you please help me to understand what should I do?

2021-04-15 16:01:19,880 UTC [26923] ERROR   Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 435, in run
    fire.Fire(ESMValTool())
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 412, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 104, in process_recipe
    recipe.run()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 1371, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 673, in run
    self._run_sequential()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 684, in _run_sequential
    task.run()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 246, in run
    input_files.extend(task.run())
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 250, in run
    self.output_files = self._run(input_files)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 473, in _run
    product.apply(step, self.debug)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 342, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 286, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 272, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/fix.py", line 118, in fix_metadata
    cube = checker(cube).check_metadata()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/check.py", line 145, in check_metadata
    self.report_errors()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/check.py", line 198, in report_errors
    raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable msftmz:
olevel: does not exist
 Coordinate basin has var name sector instead of basin
in cube:
ocean_meridional_overturning_mass_streamfunction / (kg s-1) (time: 12; -- : 3; depth: 50; latitude: 1050)
     Dimension coordinates:
          time                                                   x        -         -             -
          depth                                                  -        -         x             -
          latitude                                               -        -         -             x
     Auxiliary coordinates:
          region                                                 -        x         -             -
     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: HighResMIP
          branch_method: fixed historical forcing from 1950 was applied for a 50 year spin-up period,...
          branch_time_in_child: 18615.0
          branch_time_in_parent: 55845.0
          cmor_version: 3.3.2
          comment: Overturning mass streamfunction arising from all advective mass transport...
          data_specs_version: 01.00.23
          end_year: 2014
          experiment: coupled historical 1950-2014
          experiment_id: hist-1950
          forcing_index: 1
          frequency: mon
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.CMCC.CMCC-CM2-HR4.hist-1950.none....
          grid: ORCA0.25 1/4 deg from the Equator degrading at the poles; 1442 x 1051 longitude/latitude;...
          grid_label: gn
          initialization_index: 1
          institution: Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici, Lecce 73100,...
          institution_id: CMCC
          license: CMIP6 model data produced by CMCC is licensed under a Creative Commons...
          min_number_yrs_per_sim: 65
          mip_era: CMIP6
          nominal_resolution: 25 km
          parent_activity_id: HighResMIP
          parent_experiment_id: spinup-1950
          parent_mip_era: CMIP6
          parent_source_id: CMCC-CM2-HR4
          parent_time_units: days since 1850-1-1 00:00:00
          parent_variant_label: r1i1p1f1
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: ocean
          source: CMCC-CM2-HR4 (2016): 
aerosol: prescribed MACv2-SP
atmos: CAM4 (1deg; 288...
          source_file: /store/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/hist-1950/r1i1p1f1/Omon/msft...
          source_id: CMCC-CM2-HR4
          source_type: AOGCM
          start_year: 1950
          sub_experiment: none
          sub_experiment_id: none
          table_id: Omon
          table_info: Creation Date:(21 March 2018) MD5:74970b8e965f3034a20d7a26d3009091
          title: CMCC-CM2-HR4 output prepared for CMIP6
          variable_id: msftmz
          variant_label: r1i1p1f1
     Cell methods:
          mean: longitude (basin mean[ along zig-zag grid path])
          mean: time
2021-04-15 16:01:19,885 UTC [26923] INFO    
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

-----------------------------------RECIPE (little modifications)-----------------------------------------------------------------

# ESMValTool
# recipe_cvdp.yml
---
documentation:
  description: |
    Recipe for executing the NCAR CVDP package in the ESMValTool framework.

  authors:
    - phillips_adam
    - broetz_bjoern

  maintainer:
    - broetz_bjoern

  references:
    - phillips14eos

  projects:
    - esmval
    - c3s-magic

datasets:
#  - {dataset: CanESM2,          project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: CCSM4,            project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: CESM1-BGC,        project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: CESM1-CAM5,       project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: CESM1-FASTCHEM,   project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: CESM1-WACCM,      project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
#  - {dataset: MPI-ESM-LR,       project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 1850, end_year: 2005}
  - {dataset: CMCC-CM2-HR4, project: CMIP6, exp: hist-1950, ensemble: r1i1p1f1, grid: gn, start_year: 1950, end_year: 1950}
  - {dataset: CMCC-CM2-VHR4, project: CMIP6, exp: hist-1950, ensemble: r1i1p1f1, grid: gn, start_year: 1950, end_year: 1950}

diagnostics:

  diagnostic1:
    description: Run the NCAR CVDPackage.
    variables:
      ts:
        mip: Amon
      tas:
        mip: Amon
      pr:
        mip: Amon
      psl:
        mip: Amon
      msftmz:
        mip: Omon
    scripts:
      cvdp:
        preprocess: cmor_fix_metadata
        script: cvdp/cvdp_wrapper.py
        cvdp_run_style: serial
        cvdp_modules:
          - aice.mean_stddev
          - aice.trends_timeseries
          - amo
          - amoc
          - ipo
          - pdo
          - pr.mean_stddev
          - pr.trends_timeseries
          - psl.mean_stddev
          - psl.nam_nao
          - psl.pna_npo
          - psl.sam_psa
          - psl.trends
          - snd.mean_stddev
          - snd.trends
          - sst.indices
          - sst.mean_stddev
          - sst.trends_timeseries
          - tas.mean_stddev
          - tas.trends_timeseries
        quickplot:
          plot_type: pcolormesh

---------------------------------------------------------------Last part of main_log_debug.txt--------------------------------------

2021-04-15 16:01:19,877 UTC [26923] INFO    esmvalcore._task:125 Maximum memory used (estimate): 0.3 GB
2021-04-15 16:01:19,877 UTC [26923] INFO    esmvalcore._task:127 Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2021-04-15 16:01:19,880 UTC [26923] ERROR   esmvalcore._main:442 Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 435, in run
    fire.Fire(ESMValTool())
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 412, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 104, in process_recipe
    recipe.run()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 1371, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 673, in run
    self._run_sequential()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 684, in _run_sequential
    task.run()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 246, in run
    input_files.extend(task.run())
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 250, in run
    self.output_files = self._run(input_files)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 473, in _run
    product.apply(step, self.debug)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 342, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 286, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 272, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/fix.py", line 118, in fix_metadata
    cube = checker(cube).check_metadata()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/check.py", line 145, in check_metadata
    self.report_errors()
  File "/home/Vincenzo.DeToma/anaconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/cmor/check.py", line 198, in report_errors
    raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable msftmz:
olevel: does not exist
 Coordinate basin has var name sector instead of basin
in cube:
ocean_meridional_overturning_mass_streamfunction / (kg s-1) (time: 12; -- : 3; depth: 50; latitude: 1050)
     Dimension coordinates:
          time                                                   x        -         -             -
          depth                                                  -        -         x             -
          latitude                                               -        -         -             x
     Auxiliary coordinates:
          region                                                 -        x         -             -
     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: HighResMIP
          branch_method: fixed historical forcing from 1950 was applied for a 50 year spin-up period,...
          branch_time_in_child: 18615.0
          branch_time_in_parent: 55845.0
          cmor_version: 3.3.2
          comment: Overturning mass streamfunction arising from all advective mass transport...
          data_specs_version: 01.00.23
          end_year: 2014
          experiment: coupled historical 1950-2014
          experiment_id: hist-1950
          forcing_index: 1
          frequency: mon
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.CMCC.CMCC-CM2-HR4.hist-1950.none....
          grid: ORCA0.25 1/4 deg from the Equator degrading at the poles; 1442 x 1051 longitude/latitude;...
          grid_label: gn
          initialization_index: 1
          institution: Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici, Lecce 73100,...
          institution_id: CMCC
          license: CMIP6 model data produced by CMCC is licensed under a Creative Commons...
          min_number_yrs_per_sim: 65
          mip_era: CMIP6
          nominal_resolution: 25 km
          parent_activity_id: HighResMIP
          parent_experiment_id: spinup-1950
          parent_mip_era: CMIP6
          parent_source_id: CMCC-CM2-HR4
          parent_time_units: days since 1850-1-1 00:00:00
          parent_variant_label: r1i1p1f1
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: ocean
          source: CMCC-CM2-HR4 (2016): 
aerosol: prescribed MACv2-SP
atmos: CAM4 (1deg; 288...
          source_file: /store/CMIP6/HighResMIP/CMCC/CMCC-CM2-HR4/hist-1950/r1i1p1f1/Omon/msft...
          source_id: CMCC-CM2-HR4
          source_type: AOGCM
          start_year: 1950
          sub_experiment: none
          sub_experiment_id: none
          table_id: Omon
          table_info: Creation Date:(21 March 2018) MD5:74970b8e965f3034a20d7a26d3009091
          title: CMCC-CM2-HR4 output prepared for CMIP6
          variable_id: msftmz
          variant_label: r1i1p1f1
     Cell methods:
          mean: longitude (basin mean[ along zig-zag grid path])
          mean: time
2021-04-15 16:01:19,885 UTC [26923] INFO    esmvalcore._main:446 
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

Thanks for the attention!

valeriupredoi commented 3 years ago

hello @vincenzodetoma cheers for reporting this, it looks like CMCC-CM2 has issues with msftmz that'll need reporting and a fix, I'll transfer this issue to ESMValCore as a data issue and we'll take it from there :+1:

valeriupredoi commented 3 years ago

before I do that @phillips-ad @bjoernbroetz can you guys please give the recipe a run to reproduce the issue? :beer: EDIT: nevermind running it, the recipe is not using CMCC data as it is in master :+1: