search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
218 stars 127 forks source link

recipe_landcover not working #1292

Closed mattiarighi closed 5 years ago

mattiarighi commented 5 years ago

recipe_landcover is currently not working (tested at DKRZ). The following error is returned:

Traceback (most recent call last):
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 671, in _run_task
    output_files = task.run()
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 227, in run
    self.output_files = self._run(input_files)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 394, in _run
    product.apply(step, self.debug)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 261, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 203, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 189, in _run_preproc_function
    return function(items, **kwargs)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/fix.py", line 145, in fix_metadata
    cube = checker(cube).check_metadata()
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/check.py", line 106, in check_metadata
    self.report_errors()
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/check.py", line 124, in report_errors
    raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable baresoilFrac:
typebare: does not exist
in cube:
area_fraction / (%)                 (time: 1140; latitude: 120; longitude: 180)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Attributes:
          Conventions: CF-1.4
          associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_fx_inmcm4_rcp85_r0i0p0.nc...
          branch_time: 56940.0
          cmor_version: 2.0.0
          comment: fraction of entire grid cell  that is covered by bare soil.
          contact: Evgeny Volodin, volodin@inm.ras.ru,INM RAS, Gubkina 8, Moscow, 119333 ...
          experiment: RCP8.5
          experiment_id: rcp85
          forcing: N/A
          frequency: mon
          initialization_method: 1
          institute_id: INM
          institution: INM (Institute for Numerical Mathematics,  Moscow, Russia)
          model_id: inmcm4
          modeling_realm: land
          original_name: baresoilFrac
          parent_experiment: Historical
          parent_experiment_id: historical
          physics_version: 1
          product: output
          project_id: CMIP5
          realization: 1
          references: Volodin, Diansky, Gusev 2010. Climate model INMCM4.0. Izvestia RAS. Atmospheric...
          source: inmcm4 (2009)
          source_file: /mnt/lustre02/work/bd0854/DATA/ESMValTool2/CMIP5_ETHZ/rcp85/Lmon/bares...
          table_id: Table Lmon (12 May 2010) cde39468cd5a720342c94322249adf63
          title: inmcm4 model output prepared for CMIP5 RCP8.5
     Cell methods:
          mean: time (1 month)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_main.py", line 228, in run
    conf = main(args)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_main.py", line 156, in main
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_main.py", line 206, in process_recipe
    recipe.run()
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_recipe.py", line 1190, in run
    self.tasks, max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 596, in run_tasks
    _run_tasks_parallel(tasks, max_parallel_tasks)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 641, in _run_tasks_parallel
    task.output_files, updated_products = result.get()
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
esmvalcore.cmor.check.CMORCheckError: There were errors in variable baresoilFrac:
typebare: does not exist
in cube:
area_fraction / (%)                 (time: 1140; latitude: 120; longitude: 180)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Attributes:
          Conventions: CF-1.4
          associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_fx_inmcm4_rcp85_r0i0p0.nc...
          branch_time: 56940.0
          cmor_version: 2.0.0
          comment: fraction of entire grid cell  that is covered by bare soil.
          contact: Evgeny Volodin, volodin@inm.ras.ru,INM RAS, Gubkina 8, Moscow, 119333 ...
          experiment: RCP8.5
          experiment_id: rcp85
          forcing: N/A
          frequency: mon
          initialization_method: 1
          institute_id: INM
          institution: INM (Institute for Numerical Mathematics,  Moscow, Russia)
          model_id: inmcm4
          modeling_realm: land
          original_name: baresoilFrac
          parent_experiment: Historical
          parent_experiment_id: historical
          physics_version: 1
          product: output
          project_id: CMIP5
          realization: 1
          references: Volodin, Diansky, Gusev 2010. Climate model INMCM4.0. Izvestia RAS. Atmospheric...
          source: inmcm4 (2009)
          source_file: /mnt/lustre02/work/bd0854/DATA/ESMValTool2/CMIP5_ETHZ/rcp85/Lmon/bares...
          table_id: Table Lmon (12 May 2010) cde39468cd5a720342c94322249adf63
          title: inmcm4 model output prepared for CMIP5 RCP8.5
     Cell methods:
          mean: time (1 month)
BenMGeo commented 5 years ago

The current landcover implementation was written by @tobstac #756

tobstac commented 5 years ago

This issue is because typebare is an auxillary dimension for baresoilFract defined by the CMIP5 table. However, a number of modelers seem to have missed it and thus it is not included in all model output files. We already discussed about this last winter in #770 and at that time decided to implement a fix file for the inmcm4 model (41aeb6cf91ed386c426b57eb2330de28d2dc2094). I just updated my branch and noticed all the fix files are gone, so naturally the problem reoccurs.

tobstac commented 5 years ago

OK, I just realized the fix files did not vanish, but were moved to the esmvalcore. For some reason, Fix.get_fixes(...) does return an empty list, although the inmcm4 fix file does exists. I'll have a look and try to find out why this is happening.

tobstac commented 5 years ago

OK, found it. The issue is with esmvalcore/cmor/fixes/fix.py. The problem is all the land cover fraction variables have an uppercase F in their name, e.g. baresoilFrac. But when the fix information is converted into a dictionary, the key is turned in to a lowercase name. Thus, this can be easily fixed by replacing `variable = variable.replace('-', '')withvariable = variable.replace('-', '_').lower()` similar to what is done with project and dataset two lines above. Unfortunately, I don't know how to commit this fix, as I do find the fix.py in my anaconda environment, but not in the repository anymore.

BenMGeo commented 5 years ago

I think you have to do this in the core repository. As this is so small, you might just head to the development (edit: not master) branch there and edit the file and save to new branch instead of direct committing. (https://github.com/ESMValGroup/ESMValCore/blob/2ef34a6669cb01fa9e59f774db347c7d65942e64/esmvalcore/cmor/_fixes/fix.py#L140)

tobstac commented 5 years ago

Hm, might be a bit more complicated. According to ESMValGroup/ESMValCore#101 they omitted the variables on purpose and under the assumption all model variables are lower case. Also, these changes are not merged into the core master yet, so I can't edit it there. Maybe, @bouweandela could have a look into this, as he was involved in the issue there.

bouweandela commented 5 years ago

@tobstac I opened an issue for you, but please feel free to open an issue yourself in the ESMValCore repository if you find bugs, or even a pull request if you already know how to fix it.

mattiarighi commented 5 years ago

254 fixed this issue. Thank you @jvegasbsc! :tada: