Broken Recipes

[rjp23]

This is partly a governance question and partly a technical one.

How we're using ESMValTool in a training scenario is that we'll get it installed and working for students and then tell them to go and explore some of the existing recipes and see what they do.

Lots of these recipes don't work.

That might be because ESMValTool itself has changed and the recipe hasn't been updated but equally it might be because data has been moved or something "simple" that could be easily be fixed by someone who knows what they're doing but isn't obvious to a newbie.

Is there an expectation that every recipe provided with ESMValTool works? If not, do we need a subset of real examples (not just very basic ones) that are kept up to date and working?

My preference would be to have a few, complex, showcase examples that are guaranteed to work, e.g. the flato IPCC one is a nice example but it currently doesn't work as data has moved.

Thoughts?

[valeriupredoi]

@rjp23 can you please give some examples of recipes that don't work in the latest version installed from conda?

[valeriupredoi]

flato IPCC one is a nice example but it currently doesn't work as data has moved.

what do you mean data has moved? :beer:

[rjp23]

flato IPCC one is a nice example but it currently doesn't work as data has moved.

what do you mean data has moved? 🍺

esmvaltool -c config-user.yml recipe_flato13ipcc.yml esmvalcore._recipe_checks.RecipeError: Missing data

I don't want to debug every recipe as I don't think I have enough experience with them all to trace the errors easily but this one suggests data is missing and has been moved, right?

2020-02-24 14:33:47,545 UTC [76541] DEBUG   Found /group_workspaces/jasmin4/esmeval/obsdata-v2/
2020-02-24 14:33:47,545 UTC [76541] DEBUG   Retrieving CMIP3 configuration
2020-02-24 14:33:47,545 UTC [76541] DEBUG   Looking for files matching ['tas_*.nc'] in ['/group_workspaces/jasmin4/esmeval/obsdata-v2/']
2020-02-24 14:33:47,845 UTC [76541] INFO    Using input files for variable tas of dataset bccr_bcm2_0:

2020-02-24 14:33:47,846 UTC [76541] ERROR   No input files found for variable {'preprocessor': 'regrid_4_5', 'reference_dataset': 'ERA-Interim', 'alternative_dataset': 'NCEP', 'project': 'CMIP3', 'exp': '20c3m', 'ensemble': 'run1', 'mip': 'A1', 'field': 'T2Ms', 'start_year': 1980, 'end_year': 1999, 'variable_group': 'tas', 'short_name': 'tas', 'diagnostic': 'fig09-6_tas', 'dataset': 'bccr_bcm2_0', 'institute': 'BCCR', 'modeling_realm': 'atm', 'frequency': 'mo', 'recipe_dataset_index': 0, 'alias': 'CMIP3_bccr_bcm2_0', 'standard_name': 'air_temperature', 'long_name': 'Near-Surface Air Temperature', 'units': 'K'}
2020-02-24 14:33:47,846 UTC [76541] ERROR   Looked for files matching: /group_workspaces/jasmin4/esmeval/obsdata-v2/tas_*.nc

[rjp23]

(esmvaltool) [rparker@jasmin-sci2 esmvaltool_tests]$ esmvaltool -c config-user.yml recipe_flato_short.yml 

ValueError: Tag 'bock_ls' does not exist in section 'authors' of /home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvaltool/config-references.yml
2020-02-24 14:44:11,882 UTC [95423] INFO    If you suspect this is a bug or need help, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues and attach the run/recipe_*.yml and run/main_log_debug.txt files from the output directory.

This one is first an author name error but that's kind of my point. There should be a set that are kept up to date when something like the author name thing is changed.

Even when changing that, the data is still missing:

  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_recipe_checks.py", line 117, in data_availability
    raise RecipeError("Missing data")
esmvalcore._recipe_checks.RecipeError: Missing data

[rjp23]

esmvaltool -c config-user.yml recipe_autoassess_landsurface_soilmoisture.yml

esmvalcore._task.DiagnosticError: Diagnostic script autoassess/autoassess_area_base.py failed with return code 1. See the log in /gws/nopw/j04/ukesm_nceo/rob/esmvaltool_tests/esmvaltool_output/recipe_autoassess_landsurface_soilmoisture_20200224_144930/run/aa_landsurf_soilmoisture/autoassess_landsurf_soilmoisture/log.txt

[rjp23]

esmvaltool -c config-user.yml recipe_runoff_et.yml

2020-02-24 15:03:09,134 UTC [105611] ERROR   Program terminated abnormally, see stack trace below for more information
Traceback (most recent call last):
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_main.py", line 220, in run
    conf = main(args)
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_main.py", line 156, in main
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_main.py", line 206, in process_recipe
    recipe.run()
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_recipe.py", line 1320, in run
    max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_task.py", line 613, in run_tasks
    _run_tasks_sequential(tasks)
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_task.py", line 625, in _run_tasks_sequential
    task.run()
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_task.py", line 246, in run
    self.output_files = self._run(input_files)
  File "/home/users/rparker/miniconda2/envs/esmvaltool/lib/python3.7/site-packages/esmvalcore/_task.py", line 521, in _run
    "in {}".format(self.script, returncode, self.log))
esmvalcore._task.DiagnosticError: Diagnostic script runoff_et/catchment_analysis.py failed with return code 1. See the log in /gws/nopw/j04/ukesm_nceo/rob/esmvaltool_tests/esmvaltool_output/recipe_runoff_et_20200224_145754/run/catchment_analysis/catchment_analysis/log.txt

You get the idea! It's most of them that fail for one reason or another.

[valeriupredoi]

@rjp23 - there is a mix of issues that you are seeing because of a mix of issues with: your environment, your configuration file, using a branch that is either very old or not existing anymore: eg

• data issues:

  2020-02-24 14:33:47,545 UTC [76541] DEBUG   Retrieving CMIP3 configuration
  2020-02-24 14:33:47,545 UTC [76541] DEBUG   Looking for files matching ['tas_*.nc'] in ['/group_workspaces/jasmin4/esmeval/obsdata-v2/']
  2020-02-24 14:33:47,845 UTC [76541] INFO    Using input files for variable tas of dataset bccr_bcm2_0:

  why are you looking for CMIP3 data in obsdata-v2? -> issue with your configuration file

• recipe flato: recipe_flato_short.yml does not exist in the latest master branch

• autoassess soilmoisture should work fine now that we have the auxiliary files back on esmeval

Anyways: I would recommend pulling the latest esmvaltool and esmvalcore from master, installing them as instructed in CONTRIBUTING.md with pip and them making sure the config-user file you are using is correctly populated - can post it here and we can work together :beer:

[rjp23]

Maybe this relates to my other question about how to update things but I did (or thought I did) what you said.

I did this yesterday:

git pull origin master
conda env update --name esmvaltool --file environment.yml
conda activate esmvaltool
pip install .
python setup.py test --installation
esmvaltool --help

The flato recipe explictly looks for CMIP3 data doesn't it? i.e. it's nothing to do with my config.

https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/recipe_flato13ipcc.yml

[rjp23]

recipe_autoassess_landsurface_soilmoisture.yml

PermissionError: [Errno 13] Permission denied: '/group_workspaces/jasmin4/esmeval/autoassess_specific_files/files/ecv_soil_moisture_djf.nc'

Like I said, I didn't make this issue to start debugging every recipe but more to discuss whether it's the expectation that every packaged recipe works out of the box and if that's not guaranteed, can there be a subset that are guaranteed to work?

[valeriupredoi]

Rob, the installation is correct albeit I would avoid doing an env update - please don't read the circleci setup file but rather the instructions in CONTRIBUTING.md; you posted a failure running recipe_flato_short.yml that I don't know where you found but surely it's not in the master recipes; finding CMIP3 data has everything to do with your config file because there you specify where data lives. Like I said, if you post your config file we can help you configure it correctly :beer:

[rjp23]

I was following the readthedocs instructions:

https://esmvaltool.readthedocs.io/en/latest/getting_started/install.html

It is also possible to update an existing environment from the environment file. This may be useful when updating an older installation of ESMValTool: conda env update --name esmvaltool --file environment.yml

The recipe_flato13ipcc.yml looks for CMIP3 data but is the issue that there's not a default user config?

This is my user config file:

###############################################################################
# User's configuration file for the ESMValTool
###############################################################################
---

# Diagnostics create plots? [true]/false
write_plots: true
# Diagnositcs write NetCDF files? [true]/false
write_netcdf: true
# Set the console log level debug, [info], warning, error
log_level: debug
# verbosity is deprecated and will be removed in the future
# verbosity: 1
# Exit on warning? true/[false]
exit_on_warning: false
# Plot file format? [ps]/pdf/png/eps/epsi
output_file_type: png
# Destination directory
output_dir: ./esmvaltool_output
# Auxiliary data directory (used for some additional datasets)
auxiliary_data_dir: ./auxiliary_data
# Use netCDF compression true/[false]
compress_netcdf: false
# Save intermediary cubes in the preprocessor true/[false]
save_intermediary_cubes: true
# Remove the preproc dir if all fine
remove_preproc_dir: false
# Run at most this many tasks in parallel null/[1]/2/3/4/..
# Set to null to use the number of available CPUs.
# Make sure your system has enough memory for the specified number of tasks.
max_parallel_tasks: 1
# Path to custom config-developer file, to customise project configurations.
# See config-developer.yml for an example. Set to None to use the default
config_developer_file: null
# Get profiling information for diagnostics
# Only available for Python diagnostics
profile_diagnostic: false

# Rootpaths to the data from different projects (lists are also possible)
rootpath:
  CMIP5: /badc/cmip5/data/cmip5/output1
  CMIP6: /badc/cmip6/data/CMIP6/CMIP
  OBS: /group_workspaces/jasmin4/esmeval/obsdata-v2/
  default: /group_workspaces/jasmin4/esmeval/obsdata-v2/

# Directory structure for input data: [default]/BADC/DKRZ/ETHZ/etc
# See config-developer.yml for definitions.
drs:
  CMIP6: BADC
  CMIP5: BADC

Thanks!

[valeriupredoi]

sorry, I forgot to set group permissions to /group_workspaces/jasmin4/esmeval/autoassess_specific_files/ -> you should be able to rx it now, lemme know if still denied :beer:

[valeriupredoi]

you need to point the tool to CMIP3 in the same manner as you do for CMIP5,6:

# Rootpaths to the data from different projects (lists are also possible)
rootpath:
  CMIP3: /badc/cmip3_drs/data/cmip3/output/
  CMIP5: /badc/cmip5/data/cmip5/output1
  CMIP6: /badc/cmip6/data/CMIP6/CMIP
  OBS: /group_workspaces/jasmin4/esmeval/obsdata-v2/
  default: /group_workspaces/jasmin4/esmeval/obsdata-v2/

# Directory structure for input data: [default]/BADC/DKRZ/ETHZ/etc
# See config-developer.yml for definitions.
drs:
  CMIP3: BADC
  CMIP6: BADC
  CMIP5: BADC

[valeriupredoi]

the reason why it was looking into esmeval/obsdata-v2 is because that was (and still is) your default, and no CMIP3 path was set :beer:

[rjp23]

Thanks!

That's now looking in the right place but the data doesn't exist there.

Skipping non-existent /badc/cmip3_drs/data/cmip3/output/BCCR/bccr_bcm2_0/20c3m/mo/atm/tas/run1/{latestversion}

It should be:

/badc/cmip3_drs/data/cmip3/output/BCCR/BCM2/20c3m/

[valeriupredoi]

no, that's because your ESMValCore is not up to date - in the esmvalcore/esmvalcore/config-developer.yml you will see under CMIP3 the mapping:

  institutes:
    'bccr_bcm2_0': ['BCCR']

Rob, as I said, try and install the latest master for both ESMValTool and Core from source and not from conda - ESMValCore is out of date in this case because the conda installation of ESMValTool picks up esmvalcore from pip and there it's a bit old.

BTW all these back and forths will disappear once we've released v2 stable soon, man :beer:

[rjp23]

Thanks! I think I probably need telling exactly what commands to run as I thought I was doing that :-)

[valeriupredoi]

sure thing, let's take it offline (email on the esmvaltool list or private) - we'll fix those. Bottom line is that @mattiarighi is doing a stellar effort to make sure that with every code change the recipes work so I'd be surprised if, given correct installation and configuration of the latest code, you will find recipes that don't really work (apart from the autoassess few ones that I was a dumber and deleted the files on esmeval :stuck_out_tongue: )

[bouweandela]

Hi @rjp23 and @valeriupredoi, did you manage to find a solution for this? Did you encounter technical or documentation issues that we could solve that could help make the solution easier to find for other users?

[bouweandela]

Most recipes seem to work at the moment https://github.com/ESMValGroup/ESMValTool/issues/2535#issuecomment-1059238302. If there is a remaining issue with a specific recipe, please open a new issue.