search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

Run recipes to test ESMValCore 2.1.0 #812

Closed valeriupredoi closed 3 years ago

valeriupredoi commented 3 years ago

Hey guys @ESMValGroup/esmvaltool-developmentteam we are just about to release ESMValCore 2.1.0 on Monday next week (12 Oct 2020) and we need to make sure that all the goodies and improvements we include in the new release don't break anything in the recipes' run, could we please ask you to pick a couple of your favorite recipes and run them and let us know if there's any issue with them, by opening a GitHub issue? Please use the release branch release_2.1 - after you've run this could you pls post here which recipes you run so that other don't run them again :beer:

The reason why we ask you to do this now, and we have not in the past, is we had @mattiarighi as our testing machine, but sadly he got commandeered by other people and he's got no time to do it anymore :grin:

For installation, to be able to use the release branch, pls see the instructions to use the installation from source for ESMValCore. @jvegasbsc rightly points out in his comment that the esmvalcore version condition needs to change in the ESMValTool environment file as per https://github.com/ESMValGroup/ESMValTool/pull/1839 - note, however, that if you only reinstall ESMValCore from source to use the pre-release branch and dont need to reinstall ESMValTool, that version change is not necessary.

Note that it is not necessary to test every single recipe for the ESMValCore release, we've never done this in the past either and automatic tests are reasonably good, out of 10 or so releases it only happened once that we released an ESMValCore version that was broken (this happened because we changed things at the last minute). These tests are just to get some extra confidence that we didn't introduce any bugs.

jvegreg commented 3 years ago

There is a problem when installing release: we require <2.1. ESMValGroup/ESMValTool#1839 fixes it.

recipe_seaice_feedback.yml runs ok

valeriupredoi commented 3 years ago

cheers @jvegasbsc - good catch! An thanks for running and testing :beer:

jvegreg commented 3 years ago

recipe_albedolandcover.yml also runs ok

hb326 commented 3 years ago

recipe_eyring06jgr.yml runs ok.

hb326 commented 3 years ago

recipe_smpi.yml crashes. See attached log.

main_log.txt

@bettina-gier: did you ever have that problem?

bettina-gier commented 3 years ago

Hmm nope never seen that before, must be a new preprocessor change.

" cube.units differs: Unit('days since 1861-01-01 00:00:00', calendar='gregorian') != Unit('W m-2')" is super puzzling to me, it's trying to compare a time unit to something else? Is it because of the new change with the time handling in regrid?

valeriupredoi commented 3 years ago

that is a dummy error that masks the actual error from regridding:

Traceback (most recent call last):
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 730, in _run_task
    output_files = task.run()
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_task.py", line 242, in run
    self.output_files = self._run(input_files)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 428, in _run
    product.apply(step, self.debug)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 296, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 241, in preprocess
    result.append(_run_preproc_function(function, item, settings))
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/__init__.py", line 224, in _run_preproc_function
    return function(items, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid.py", line 335, in regrid
    cube = esmpy_regrid(cube, target_grid, scheme)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 331, in regrid
    regridder = build_regridder(src_rep, dst_rep, method)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 220, in build_regridder
    regridder = build_regridder_2d(src_rep, dst_rep,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 149, in build_regridder_2d
    dst_field = cube_to_empty_field(dst_rep)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 130, in cube_to_empty_field
    grid = get_grid(*esmpy_coords, circular=circular)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 88, in get_grid
    grid = ESMF.Grid(np.array(esmpy_lat.shape),
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/ESMF/util/decorators.py", line 64, in new_func
    return func(*args, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/ESMF/api/grid.py", line 385, in __init__
    self._struct = ESMP_GridCreateNoPeriDim(self.max_index,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/ESMF/interface/cbindings.py", line 560, in ESMP_GridCreateNoPeriDim
    raise ValueError('ESMC_GridCreateNoPeriDim() failed with rc = '+str(rc)+
ValueError: ESMC_GridCreateNoPeriDim() failed with rc = 545.    Please check the log files (named "*ESMF_LogFile").

there is a serious issue regridding MRI-CGM3 - @zklaus any clues?

valeriupredoi commented 3 years ago

OK ran it three times for MRI-CGCM3/hfds and it succesfully completed the preprocessing, @bettina-gier @hb326 could you guys pls run it again? The ESMPY error was a fluke, hopefully your issue was too, I see no reason why it should fail :beer:

bouweandela commented 3 years ago

From the log, it looks like a problem with some model when loading the data, a coordinate is misinterpreted as a cube, probably because some attribute is missing somewhere in the netcdef file and then it's trying to merge that coordinate and the cube with the variable into a single cube, which fails for obvious reasons.

@hb326 Could you please attach the file main_log_debug.txt instead? Then we might be able to figure out what model it is that is causing the problem.

valeriupredoi commented 3 years ago

it's MRI-CGCM3/hfds which is not problematic with data off CEDA ESGF on Jasmin, might be the data off the DKRZ ESGF node

hb326 commented 3 years ago

Sure. Here it is:

main_log_debug.txt

valeriupredoi commented 3 years ago

all autoassess recipes (bar the unreviewed ones), all validation, and all _ocean* recipes run with no problems (@ledm owes me a :beer: for this but that's okay, I think he owes me about a dozen so far :grin: )

bouweandela commented 3 years ago

@hb326 Could you also share the recipe that you're running? From the log it looks like it's trying to regrid hfds from model MRI-CGCM3 to the grid of model GFDL-ESM2G, while the SMPI recipe in the ESMValTool actually has a target grid 1x1.

hb326 commented 3 years ago

I run the recipe as it comes in the master branch: https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/recipe_smpi.yml

bascrezee commented 3 years ago

I thought I also quickly test recipe_smpi.yml from the master branch. I am testing with master branch of ESMValTool and release_2.1 from ESMValCore (just checkin). This is the error message I get, hope it helps. 

 in memory consumption occur.
2020-10-07 14:13:26,492 UTC [36998] ERROR   Program terminated abnormally, see stack trace below for more information
Traceback (most recent call last):
  File "/home/crezees/ESMValCore/esmvalcore/_main.py", line 430, in run
    fire.Fire(ESMValTool())
  File "/net/exo/landclim/crezees/conda/envs/esmvaltool-pub/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/net/exo/landclim/crezees/conda/envs/esmvaltool-pub/lib/python3.8/site-packages/fire/core.py", line 463, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/net/exo/landclim/crezees/conda/envs/esmvaltool-pub/lib/python3.8/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/crezees/ESMValCore/esmvalcore/_main.py", line 407, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/crezees/ESMValCore/esmvalcore/_main.py", line 94, in process_recipe
    recipe = read_recipe_file(recipe_file, config_user)
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 71, in read_recipe_file
    return Recipe(raw_recipe,
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 951, in __init__
    self.tasks = self.initialize_tasks() if initialize_tasks else None
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 1299, in initialize_tasks
    task = _get_preprocessor_task(
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 917, in _get_preprocessor_task
    task = _get_single_preprocessor_task(
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 776, in _get_single_preprocessor_task
    products = _get_preprocessor_products(
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 733, in _get_preprocessor_products
    _update_target_grid(
  File "/home/crezees/ESMValCore/esmvalcore/_recipe.py", line 198, in _update_target_grid
    parse_cell_spec(settings['regrid']['target_grid'])
  File "/home/crezees/ESMValCore/esmvalcore/preprocessor/_regrid.py", line 87, in parse_cell_spec
    raise ValueError(emsg.format(spec))
ValueError: Invalid MxN cell specification for grid, got 'reference_dataset'.
valeriupredoi commented 3 years ago

I run the recipe as it comes in the master branch: https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/recipe_smpi.yml

@hb326 @bascrezee you guys must pull the latest ESMValTool master, I have this for hfds preproc:

 ppNOLEVirreg:
    regrid:
      target_grid: 1x1
      scheme: linear

It works fine for me (well, apart from that ESMPY transitory error)

hb326 commented 3 years ago

I just pulled the latest version of the master branch, and the recipe is still the same as before... Maybe we have to adjust the recipe?

katjaweigel commented 3 years ago

recipe_deangelis15nat.yml and recipe_li17natcc.yml run ok. Thanks @jvegasbsc for the fix, didn't understand how to do it without.

schlunma commented 3 years ago

I added a checklist with all recipes of the current master to the initial post by @valeriupredoi so we get a nice overview which recipe hasn't been tested yet. If someone intends to do tests but is not finished he/she could add his/her name to the corresponding recipe so that we avoid duplicate efforts.

bouweandela commented 3 years ago

Note that it is not necessary to test every single recipe for the ESMValCore release, we've never done this in the past either and automatic tests are reasonably good, out of 10 or so releases it only happened once that we released an ESMValCore version that was broken (this was because I changed things at the last minute). These tests are just to get some extra confidence that we didn't introduce any bugs.

schlunma commented 3 years ago

Ah sorry, my mistake. I thought we tested all recipes for v2.0 and have to do this again for v2.1

bouweandela commented 3 years ago

Yes, we will test everything again, but that will be in two weeks time for the release of the tool. But it's great that you added an overview, that's still really useful!

bouweandela commented 3 years ago

@hb326 @bascrezee I have opened a new issue for the problem with recipe_smpi.yml

katjaweigel commented 3 years ago

recipe_martin18grl.yml and recipe_spei.yml are running

soufianekar commented 3 years ago

recipe_cvdp working fine

valeriupredoi commented 3 years ago

cheers muchly everyone! and thanks for the graph list @schlunma :beer: Keep up the momentum, we are gaining on the enemy (no enemy per se, but I thought if I sounded like Gen. Patton you might run more recipes :rofl: )

schlunma commented 3 years ago

Succesfully tested recipe_ecs.yml, recipe_cox18nature.yml, recipe_tcr.yml and recipe_ecs_constraints.yml.

Peter9192 commented 3 years ago

Representing the hydrology section with a report on recipe_pcrglobwb.yml: :heavy_check_mark:

remi-kazeroni commented 3 years ago

Successfully tested recipe_consecdrydays.yml, recipe_ensclus.yml and recipe_zmnam.yml .

dr-ko commented 3 years ago

recipe_carvalhais14nat is running. 🍻

zechlau commented 3 years ago

recipe_wenzel16nat is running too

valeriupredoi commented 3 years ago

hey guys, cheers muchly for doing all this work, @bouweandela and myself have now released version 2.1.0 - next stop, ESMValTool v2.1.0 when we'll really have to run all them recipes :beer: (code freeze next Monday, 19 Oct for ESMValTool, testing that week)