search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
224 stars 128 forks source link

esmvaltool does not find/recognise ERA-Interim input files #741

Closed riddernina closed 5 years ago

riddernina commented 5 years ago

I have been trying to feed ERA-Interim into esmvaltool to run the surge estimator, however I keep getting an error message (see below). I assume that this means that the input files are not found. I made sure that the path to OBS in the config-user file are correct and have tried various name set ups for the input file without knowing what the actual name should look like. I also moved the input files to other directories (changing the config file accordingly) but no luck.

How can I make sure that esmvaltool finds the input?

Thanks in advance for you help! Cheers, Nina

2018-12-05 08:48:54,873 UTC [22172] INFO esmvaltool._main:127 

      _____ ____  __  ____     __    _ _____           _
     | ____/ ___||  \/  \ \   / /_ _| |_   _|__   ___ | |
     |  _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
     | |___ ___) | |  | | \ V / (_| | | | | (_) | (_) | |
     |_____|____/|_|  |_|  \_/ \__,_|_| |_|\___/ \___/|_|

ESMValTool - Earth System Model Evaluation Tool.

http://www.esmvaltool.org

CORE DEVELOPMENT TEAM AND CONTACTS: Veronika Eyring (PI; DLR, Germany - veronika.eyring@dlr.de) Bouwe Andela (NLESC, Netherlands - b.andela@esciencecenter.nl) Bjoern Broetz (DLR, Germany - bjoern.broetz@dlr.de) Niels Drost (NLESC, Netherlands - n.drost@esciencecenter.nl) Nikolay Koldunov (AWI, Germany - nikolay.koldunov@awi.de) Axel Lauer (DLR, Germany - axel.lauer@dlr.de) Benjamin Mueller (LMU, Germany - b.mueller@iggf.geo.uni-muenchen.de) Valeriu Predoi (URead, UK - valeriu.predoi@ncas.ac.uk) Mattia Righi (DLR, Germany - mattia.righi@dlr.de) Javier Vegas-Regidor (BSC, Spain - javier.vegas@bsc.es)

For further help, please read the documentation at http://esmvaltool.readthedocs.io. Have fun!

2018-12-05 08:48:54,873 UTC [22172] INFO esmvaltool._main:129 Using config file /usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/config-userNR.yml 2018-12-05 08:48:54,873 UTC [22172] INFO esmvaltool._main:130 Writing program log files to: /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/run/main_log.txt /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/run/main_log_debug.txt 2018-12-05 08:48:54,873 UTC [22172] INFO esmvaltool._main:158 Starting the Earth System Model Evaluation Tool v2.0a1 at time: 2018-12-05 08:48:54 UTC 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:160 ---------------------------------------------------------------------- 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:161 RECIPE = /usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/recipes/recipe_surgeheight.yml 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:162 RUNDIR = /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/run 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:163 WORKDIR = /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/work 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:164 PREPROCDIR = /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/preproc 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:165 PLOTDIR = /usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/plots 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:166 ---------------------------------------------------------------------- 2018-12-05 08:48:54,874 UTC [22172] INFO esmvaltool._main:169 Running tasks using at most 1 processes 2018-12-05 08:48:54,875 UTC [22172] INFO esmvaltool._main:172 If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. In that case, try reducing 'max_parallel_tasks' in your user configuration file. 2018-12-05 08:48:54,875 UTC [22172] DEBUG esmvaltool._recipe:85 Checking recipe against schema /usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/recipe_schema.yml 2018-12-05 08:48:54,894 UTC [22172] DEBUG esmvaltool._recipe:834 Retrieving diagnostics from recipe 2018-12-05 08:48:54,894 UTC [22172] DEBUG esmvaltool._recipe:916 Populating list of variables for diagnostic surge_estimator 2018-12-05 08:48:54,894 UTC [22172] DEBUG esmvaltool._config:171 Retrieving institutes for dataset ERA-Interim 2018-12-05 08:48:54,894 UTC [22172] DEBUG esmvaltool._config:163 Retrieving OBS configuration 2018-12-05 08:48:54,894 UTC [22172] DEBUG esmvaltool._config:171 Retrieving institutes for dataset ERA-Interim 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._config:163 Retrieving OBS configuration 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._config:171 Retrieving institutes for dataset ERA-Interim 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._config:163 Retrieving OBS configuration 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._recipe:937 Setting script for diagnostic surge_estimator 2018-12-05 08:48:54,895 UTC [22172] INFO esmvaltool._recipe:994 Creating tasks from recipe 2018-12-05 08:48:54,895 UTC [22172] INFO esmvaltool._recipe:998 Creating tasks for diagnostic surge_estimator 2018-12-05 08:48:54,895 UTC [22172] INFO esmvaltool._recipe:1003 Creating preprocessor task surge_estimator/psl 2018-12-05 08:48:54,895 UTC [22172] INFO esmvaltool._recipe:742 Creating preprocessor 'preprocessor1' task for variable 'psl' 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._recipe:268 If not present: adding keys from CMOR table to {'preprocessor': 'preprocessor1', 'field': 'T2D', 'short_name': 'psl', 'diagnostic': 'surge_estimator', 'dataset': 'ERA-Interim', 'project': 'OBS', 'type': 'reanalysis', 'version': 1, 'tier': 3, 'start_year': 1979, 'end_year': 1980, 'cmor_table': 'OBS', 'filename': '/usr/people/ridder/nobackup_1/esmvaltool_output/recipe_surgeheight_20181205_084854/preproc/surge_estimator_preprocessor1_psl/OBS_ERA-Interim_reanalysis_1_T2D_psl_1979-1980.nc'} 2018-12-05 08:48:54,895 UTC [22172] DEBUG esmvaltool._recipe:271 Skipping because cmor_table or mip not specified 2018-12-05 08:48:55,876 UTC [22172] ERROR esmvaltool._main:223 Program terminated abnormally, see stack trace below for more information Traceback (most recent call last): File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_main.py", line 215, in run conf = main(args) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_main.py", line 143, in main process_recipe(recipe_file=recipe, config_user=cfg) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_main.py", line 189, in process_recipe recipe = read_recipe_file(recipe_file, config_user) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 56, in read_recipe_file raw_recipe, config_user, initialize_tasks, recipe_file=filename) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 816, in init self.tasks = self.initialize_tasks() if initialize_tasks else None File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 1008, in initialize_tasks write_ncl_interface=self._support_ncl) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 794, in _get_preprocessor_task variables, profile, config_user, ancestors=derive_tasks) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 705, in _get_single_preprocessor_task variables=variables, profile=profile, config_user=config_user) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 637, in _get_preprocessor_settings settings = _get_default_settings(variable, config_user, derive=derive) File "/usr/people/ridder/Documents/0_models/ESMValTool/esmvaltool/_recipe.py", line 439, in _get_default_settings settings['load_cubes']['constraints'] = variable['standard_name'] KeyError: 'standard_name'

jhardenberg commented 5 years ago

Hello Nina, I notice that your recipe specifies type: reanalysis for the ERA-Interim dataset, I believe that should now be type: reanaly . Maybe could you try this way?

riddernina commented 5 years ago

Unfortunately that doesn't solve the problem. Thanks for the suggestion though!

bjoernbroetz commented 5 years ago

Your log terminates with:

KeyError: 'standard_name'

There is currently a discussion about non derived variables going on in #725 . Could you update your current branch with the latest version2_development branch and send again the error message if it fails?

bjoernbroetz commented 5 years ago

You will also need to update your environment

bouweandela commented 5 years ago

Assuming we are talking about this recipe: https://github.com/ESMValGroup/ESMValTool/blob/MAGIC_surge_height/esmvaltool/recipes/recipe_surgeheight.yml

There are various issues:

Regarding the filenames, in my input data directory, ERA interim data is stored with names like: Tier3/ERA-Interim/OBS_ERA-Interim_reanaly_1_T2Ds_psl_197901-201412.nc

bouweandela commented 5 years ago

Regarding the mip key, I have never worked with cmor tables and have no clue about the standards that you have to declare in your netCDF files. I tried to copy the format and keys from the files that I had, i.e. the historical run form the Canadian ensemble, so according to this I added the mip key ‘6hPlev’ to the ERA-Interim inputfile itself. I assume I have to add this to the recipe as well for it to work.

There is no need to add the mip key to the netcdf file. You should just put it in the recipe. CMOR tables are used to check that your data complies with the cf standard, e.g. has the correct names for the variable, dimensions, units and the values are within the valid range. Version 1 of esmvaltool includes a not so easy to use reformat script for ERA-Interim data (I'm not sure what the status in version 2 is), but maybe @bjoernbroetz can also provide you with the correct data.

I had a look at the cmor tables, and the only CMIP5 tables that include both psl and uas/vas are esmvaltool/cmor/tables/cmip5/Tables/CMIP5_day esmvaltool/cmor/tables/cmip5/Tables/CMIP5_Amon esmvaltool/cmor/tables/cmip5/Tables/CMIP5_cfSites esmvaltool/cmor/tables/cmip5/Tables/CMIP5_cf3hr so that means you probably need to choose a mip from the list day, Amon, cfSites, cf3hr.

riddernina commented 5 years ago

Perfect that solved it! Thank you very much!