Closed remi-kazeroni closed 2 years ago
The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc2/ Among the 119 available recipes, 80 were ran successfully and 39 failed. A summary of the main failure reasons will be posted soon in this issue.
@ESMValGroup/esmvaltool-recipe-maintainers
Please check that your recipes ran successfully using the link above and let us know something needs to be corrected before the release. This includes that all plots are created and look fine, output files look fine, provenance is stored (checklist here and here).
Everyone can also help us by running a few recipes with the latest development installation in order to confirm these results.
Settings for the recipe testing
Most recipes were run using the Cylc suite using max_parallel_tasks: 8
, a time limit of 4h, and 16 GB of RAM. The more memory-intensive recipes were run on a full prepost node. All recipes were run on Mistral using offline=False
to automatically download missing data. A total of 110 GB was downloaded (59 GB of CMIP6 data and 52 GB of CMIP5 data). We would need to understand why these data are not available on Mistral. This goes beyond the recent deletion of some CMIP5 data.
And special thanks to @bouweandela for making the recipe testing much more convenient 🥳
Edit: I reran 4 recipes using search_connection:
url: "http://esgf-index1.ceda.ac.uk/esg-search"
instead of url: http://esgf-data.dkrz.de/esg-search
but that didn't help. The search through ESGF nodes takes more than the time limit (12 hours) or the data download still fails. These recipes are: mpqb/recipe_mpqb_xch4.yml
, recipe_russell18jgr.yml
, bock20jgr/recipe_bock20jgr_fig_8-10.yml
, recipe_deangelis15nat.yml
.
Below we summarize all the issues found by the automatic run. Feel free to expand this table as you see fit.
These issues will not be fixed in this release. They are generally due to data that has become unavailable or is not yet fully integrated into ESMValTool.
Recipe | Missing variables | Known problem? | Current Status |
---|---|---|---|
recipe_check_obs | rlns, rlus, rsns, rsus | #1388 |
Recipe | Reason of failure | Known problem? | Current Status |
---|---|---|---|
recipe_collins13ipcc | missing CMIP5 data (areacello, rtmt, tas) | #2408 (but more missing); Birgit: I commented out the models for which the "tas" and "rtmt" data were missing, I did not change anything related to the missing "areacello" data; an updated version of the recipe is available in the "fix_recipes_with_missing_data_branch" | |
recipe_flato13ipcc | missing CMIP5 data (areacello) | #2408 | |
recipe_landcover | missing CMIP5 data | #2408; Birgit: I commented out the model for which the data is not available anymore; the updated recipe is available in the "fix_recipes_with_missing_data_branch" | |
recipe_recipe_perfmetrics_CMIP5 | missing CMIP5 data (CESM1-CAM5-1-FV2: tas) | #2408, Birgit: I commented out the one model for "tas" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch" | |
recipe_recipe_perfmetrics_CMIP5_4cds | missing CMIP5 data (CESM1-CAM5-1-FV2: tas) | #2408; Birgit: I commented out the one model for "tas" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch" | |
recipe_recipe_perfmetrics_CMIP5_land | missing CMIP5 data (BNU-ESM: fgco2) | #2408; Birgit: I commented out the one model for "fgco2" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch" | |
recipe_seaice | missing CMIP5 data (areacello) | #2408 (but more missing) | |
recipe_wenzel14jgr | missing CMIP5 data (fgco2, nbp) | #2408 (but more missing); Birgit: I commented out the missing models in the recipe; there were already some models commented out, they are now marked with a double "##"; an updated version of the recipe is available in the "fix_recipes_with_missing_data_branch" | |
recipe_williams09climdyn | missing CMIP5 data (sic) | #2408; Birgit: 4 out of 5 datasets are not available; commenting all four out seems kind of wrong since then only one model (+ one CMIP6 model) would be left over to be used in the diagnostic; note by @axel-lauer: I think I vaguely remember that sic from CMIP5 AMIP runs was only available for very few models. As sic is prescribed in these runs, we used the same fields (from one model) for all others. So I guess this never worked in v2.0 and has to be remodeled. I would say let's postpone this for now. |
Recipe | Reason of failure | Known problem? | Current Status |
---|---|---|---|
recipe_climwip_brunner20esd | missing CESM2 data (tas) | Birgit: CESM2 data is available (on MISTRAL) but the specified ensembles for the ssp585 scenario are not; ensembles are specified as "ensemble: r(1:2)i1p1f1", but only r4i1p1f1, r10i1p1f1 and r11i1p1f1 are available |
Recipe | Reason of failure | Known problem? | Current Status |
---|
Recipe | Reason of failure | Known problem? | Current Status |
---|---|---|---|
recipe_bock20jgr_fig_8-10 | ESGF data request not finished after 12 hours | Used to work before partial deletion of CMIP5 data at DKRZ | |
recipe_impact | data download failure (EC-Earth3-Veg, areacella) | ||
recipe_russell18jgr | ESGF data request failure (twice the same) |
Recipe | Reason of failure | Known problem? | Current Status |
---|
Recipe | Reason of failure | Known problem? | Current Status |
---|
Recipe | Reason of failure | Known problem? | Current Status |
---|---|---|---|
recipe_extreme_events | cdo remapcon |
Recipe | Reason of failure | Known problem? | Current Status |
---|
The overview of issues found by running all recipes automatically is now complete (see post above). Please add already opened issues to the tables, we may have missed some.
It seems that the missing CMIP5 issue has slightly worsened since the last release (see ESMValGroup/ESMValTool#2408). But we were told that some deleted data will be retrieved from the archive soon.
We also have a few failures with the automatic download features (too long request on ESGF nodes, download failures, existing data but not found). Could someone else try to rerun some of the impacted recipes? I tried myself but got the same failures. I may not be using optimal settings for the automatic download. Any suggestion welcomed!
stellar work, guys! What's the actual problem with the two Julia recipes? For recipe_autoassess_landsurface_soilmoisture
we have https://github.com/ESMValGroup/ESMValTool/issues/2309 - but should work on JASMIN where those files currently live, I'll have a test now :+1:
also, I'm gonna run a bunch of those CMIP5 missing recipes since we have CMIP5 on JASMIN (unlike some German HPC that did a terribly ungerman thing and deleted all that data :rofl: )
OK here's me stuffs on JASMIN:
recipe_anav13jclim.yml
: missing a LOT of of OBS and two CMIP5 (even if I turn on auto dll): CMIP5_inmcm4: cSoil and CMIP5_inmcm4: cVegrecipe_flato13ipcc.yml
- missing OBS, CMIP3 and areacello fx files for abrupt mip - a total data messrecipe_autoassess_landsurface_soilmoisture.yml
- ran OK (phew)examples/recipe_julia.yml
- ran OKrecipe_rainfarm.yml
- ran OKlet me know if you guys want me to run any others but bugger me flat, am missing lots of OBS's at JASMIN :mag:
What's the actual problem with the two Julia recipes?
Here is the type of error I am getting. Did I do something wrong when installing Julia? I simply followed the instructions and did esmvaltool install Julia
which was successful.
you need to purge the (most probably existing) ~/.julia
dir, re-create the env so Julia can be installed cleanly from conda-forge
, install esmvaltool, then esmvaltool install Julia
We had two cases where using offline=true
instead of offline=false
was enough to run the recipe successfully: recipe_deangelis15nat.yml
and recipe_mpqb_xch4.yml
. In both cases, all the files needed are available in the official pool of CMIP5-6 data on Mistral and no data download is needed. Nevertheless, if the automatic download option is used, the tool will search for the many missing fx
files on ESGF nodes in order to download these files which are later not used in the recipe. In particular the search for fx
files took more than 12 hours (wall time on Mistral) for recipe_deangelis15nat.yml
while the run with offline=true
took 1 hour. Would this deserves a separate issue for further discussion?
yes - very good find, Remi! That's a pretty serious performance bug :bug:
I agree that this is definitely something we need to check, but we probably can't fix this in time for v2.5. I suggest adding this to v2.6.
am opening an issue as we speak
also, I ran recipe_seaice.yml
on JASMIN:
2022-02-16 16:48:03,524 UTC [20412] ERROR Missing data for preprocessor seaice_tsline/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR Missing data for preprocessor seaice_trends/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR Missing data for preprocessor seaice_ecs/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_historical_r0i0p0: areacello
- Missing data for CMIP5_bcc-csm1-1-m_rcp85_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_historical_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR Missing data for preprocessor seaice_yod/areacello:
- Missing data for bcc-csm1-1-m_historical: areacello
- Missing data for inmcm4_historical: areacello
this is like shooting ducks with ballistic missiles: MRI has areacello
in historical
only but BCC has the same fx variable in piControl
only - if we don't get the wildcard search in we'll always hit these sort of roadblocks; esgf-pyclient has no chance to retrieve it either if the same crappe is at other ESGF nodes too, it will just end up spending time looking around
@ESMValGroup/esmvaltool-recipe-maintainers Please check that your recipes ran successfully using the link above and let us know something needs to be corrected before the release. This includes that all plots are created and look fine, output files look fine, provenance is stored (checklist here and here).
Should we do a comment with the list of full recipes and a checkbox for having them checked for the scientific ok - provenance and figures looking fine? Otherwise we never know if anyone actually looked at the output for the recipes that ran fine.
@bettina-gier yes that is a good idea always - maybe this can be done during the freeze, like we did during other releases :beer:
I just tried to identify the issue with recipe_smpi_4cds.yml
. I think this might be related to memory issues: when using only the years 2001-2005 the diagnostic ua
(all other diagnostics deactivated) the recipe runs fine, when using 2000-2005 (or the original range 1986-2005) the following error appears:
Traceback (most recent call last):
...
File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 468, in multi_model_statistics
group_statistics = _multiproduct_statistics(
File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 370, in _multiproduct_statistics
statistics_cubes = _multicube_statistics(cubes=cubes,
File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 351, in _multicube_statistics
result_cube = _compute_eager(aligned_cubes,
File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 286, in _compute_eager
single_model_slices = [cube[chunk] for cube in cubes]
File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 286, in <listcomp>
single_model_slices = [cube[chunk] for cube in cubes]
File "/work/bd0854/b309192/soft/miniconda3/envs/core_rc2/lib/python3.10/site-packages/iris/cube.py", line 2323, in __getitem__
dimension_mapping, data = iris.util._slice_data_with_keys(
File "/work/bd0854/b309192/soft/miniconda3/envs/core_rc2/lib/python3.10/site-packages/iris/util.py", line 779, in _slice_data_with_keys
raise IndexError("Cannot index with zero length slice.")
The weird thing is that I'm running this on a 256 GB node, and the output says Maximum memory used (estimate): 48.4 GB
. Note that this is 4D (time, plev, lat, lon) data, so memory issues would not be surprising.
no, it says Maximum memory used (estimate): 187.4 GB - but don't matter, if it was memory it would have said MemoryError
:grin: It looks like _compute_slices()
returns 0-len slices, it could be that n_slices = int(np.ceil(total_bytes / gigabyte))
is pish - something with too little memory estimated?
Yes, you're right!! It is related to this. Here is the output of some prints:
n_timesteps: 71
slices_len: 8
n_slices: 10
start, end: 0 8
start, end: 8 16
start, end: 16 24
start, end: 24 32
start, end: 32 40
start, end: 40 48
start, end: 48 56
start, end: 56 64
start, end: 64 71
start, end: 72 71
The last indices are definitely not correct. I will try to fix this.
(the 48.4 GB referred to my run with only 6 year; the 187.4 GB correspond to using all 20 years)
Status of the release process and recipe testing:
Using the release candidate v2.5rc2 for the Core, we initially got 80 of 119 public recipes to run successfully during the automated testing. Over the past week, the number of successful recipes was increased to 96 by:
The latest fixes to the Core warrants another (an probably last) release candidate for the Core. @schlunma will soon release v2.5rc3. This will be done at the same time as the feature freeze for the Tool. We will perform another round of recipe testing and ask maintainers to check the output later this week.
Recipe testing against v2.5.0rc3 of the Core
The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc3/ Among the 120 available recipes, 96 were ran successfully and 24 failed.
Given the current uncertainties in the release process and the choice of iris version used for v2.5 (see https://github.com/ESMValGroup/ESMValCore/issues/1509), we may not need to ask recipe maintainers to check the results of their recipes at this stage. We will do so for the next round of recipe testing (most likely needed if iris=3.1 is chosen) or contact maintainers to check this round of recipe testing later on. Nevertheless, feel all free to take a look at your favourite recipes.
Summary of the main issues
6 recipes ran successfully with v2.5.0rc2 but not this time:
recipe_anav13jclim
, recipe_climwip_brunner2019_med
, recipe_schlund20esd
=> trouble to find all data needed (we have had some issues with CMIP5 data availability at DKRZ recently)
schlund20jgr/recipe_schlund20jgr_gpp_*
(all three recipes)
=> diagnostic issue related to https://github.com/ESMValGroup/ESMValCore/issues/1509
Among the 18 other failing recipes, 1 "new" issue discovered:
recipe_impact
(data issue at preprocessing stage)
=> https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc3/recipe_impact_20220223_103317/
In the previous round, the recipe didn't finish looking for data through ESGF nodes.The 17 other failures remain unchanged:
recipe_autoassess_landsurface_soilmoisture_20220222_161348
and recipe_autoassess_radiation_rms_cfMon_all_20220223_082002
=> still not running on Mistral, better to test them on Jasminrecipe_julia
and recipe_rainfarm
=> still an issue for me to run Julia recipes...recipe_check_obs
=> known in https://github.com/ESMValGroup/ESMValCore/issues/1388recipe_climwip_brunner20esd
, recipe_collins13ipcc
, recipe_flato13ipcc
, recipe_perfmetrics_CMIP5*
(all 3), recipe_seaice
, recipe_wenzel14jgr
, recipe_williams09climdyn_CREM
=> missing data issuerecipe_bock20jgr_fig_8-10
=> ESGF search takes longer than job wall time (see https://github.com/ESMValGroup/ESMValCore/issues/1495).recipe_extreme_events
=> diagnostic (cdo) issue, see tables above.recipe_carvalhais14nat
=> Matplotlib (norm, vmin, vmax) error (@schlunma, not sure how you got this running last time).I wonder if we could speed-up the search on ESGF nodes for some recipes (recipe_anav13jclim
, recipe_climwip_brunner2019_med
, recipe_schlund20esd
, recipe_bock20jgr_fig_8-10
, but not only) by following this suggestion from @bouweandela regarding areacella
. Of course, this wouldn't not apply to all cases, but it could help to save computing resources.
[x] recipe_carvalhais14nat
: I just found out that I was using matplotlib=3.4.3
, but the error only appears in matplotib=3.5.1
. I will have a look at that. -> #2558
[x] recipe_impact
: That looks like a fix is needed for the dataset. I will also have a look at that. -> #2563
[x] recipe_schlund20esd
: This one already has fx_variables: areacella
, but needs LOTS of data. Not sure about the other recipes that ran into a timeout.
[x] recipe_anav13jclim
: does not use area_statistics
, so it doesn't suffer from the areacella
/areacello
issue
cheers @remi-kazeroni and @schlunma - for the Julia ones, have you nuked the presumably existent ~/.julia
dir and still they don't run properly (after a reinstall with esmvaltool install Julia
)? Yes, the Autoassess recipes run fine on JASMIN :+1:
The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc4/debug.html You can also have a look at the recipe portal to browse these results: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc4/
Among the 121 available recipes, 111 were run successfully and only 10 failed. This is the result of fixes to the Core, recipes/diagnostics and chasing missing data. A summary of the remaining failures in the next post.
This last round of recipe testing also shows nice improvement in the automatic download of data thanks to the updated esgf-pyclient package. The data search through ESGF nodes takes only minutes in cases where it used to take hours to find all the fx files (see https://github.com/ESMValGroup/ESMValCore/issues/1495). Thanks for your great work @bouweandela!
@ESMValGroup/esmvaltool-recipe-maintainers, it would be great if you could check the output of the recipes you are responsible for and tick the corresponding box if happy with the output (checklist here and here). If something does not look right, please let us know if this issue.
List of recipes:
Summary of the main issues
CMIP6_CESM2_r1i1p1f1: tas
and CMIP6_CESM2_r2i1p1f1: tas
) fixed in #2581 excellent work guys! I'll test the AA recipes on JASMIN, so we have only 8 crappe'd :grin:
the cfMon_all
AA recipe dies with the usual
esmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp:
Coordinate plevs has var name plev7 instead of plev
tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not None
clisccp: does not match coordinate rank
in cube:
issue coz of the ISCCP data on JASMIN - I can turn that dataset off, but I know for a fact that that should work if we fix the ISCCP data, what do you recommend @remi-kazeroni ? The other AA went through no problemo
the
cfMon_all
AA recipe dies with the usualesmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp: Coordinate plevs has var name plev7 instead of plev tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not None clisccp: does not match coordinate rank in cube:
issue coz of the ISCCP data on JASMIN - I can turn that dataset off, but I know for a fact that that should work if we fix the ISCCP data, what do you recommend @remi-kazeroni ? The other AA went through no problemo
Thanks for double checking the AA recipes @valeriupredoi! We already have an issue open for the problematic dataset: https://github.com/ESMValGroup/ESMValCore/issues/1238 If it's better to keep the dataset in the recipe, I'd suggest to address that for the next release 👍
awesome, yeah - I believe I opened that issue judging by the title containing semi-expletives :sweat_smile:
Status of the recipe testing using the version v2.5.0rc4 for the Core: there are up to 115 recipes that can be run successfully. Among the remaining 6 recipes, there should not be any missing data problem any more (CMIP5, CMIP6, OBS,...). For each failing recipe, there is a dedicated issue or PR to fix it. In the (hopefully 😆) last round of testing, we need to pay closer attention to the recipe_perfmetrics_land_CMIP5
(model fix needed?) and the recipe_collins13ipcc
. For the later one, there is probably not enough memory on the Mistral nodes to run it. But maybe it could work with Levante, worth trying.
@remi-kazeroni stellar work! Do we have a mention in the documentation of that recipe that needs gargantuan amounts of memory?
For information, our ESMValTool project data from Mistral will be copied to Levante (successor machine) on March 14. This comprises user data (e.g. /work/bd0854/b3*****) but also OBS/RAWOBS data and data from different projects. After the copy, DKRZ will try to keep the data synchronize between the 2 machines until March 22. After that, it will be our responsibility to copy data from Mistral to Levante. More info here: https://docs.dkrz.de/blog/2022/mistral-levante-data-move.html
I think this will require some actions from us to adjust the data paths, reorganize the directory structure of the group space... But for the time being, I'm hoping that we could finish the recipe testing and the release on Mistral only and configure things on Levante once the release is over. So hopefully we can finish the release sometime next week at the latest :)
The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc6/debug.html You can also have a look at the recipe portal to browse these results: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc6/
Among the 121 available recipes, 114 were run successfully and only 7 failed. Since the outcome of this round of testing is identical to the previous one, there is no need for another thorough recipe output checking. Nevertheless, feel free to check the output of your favourite recipe if you wish. I think this round of testing is good enough to do the release next week 🍻 No new bug was discovered during this testing. Among the 7 known issues, we have:
multi_model_statistics
step, see https://github.com/ESMValGroup/ESMValTool/issues/2594make that 6 now :grin: so that's a 5% loss ratio - am very happy and it's a good start for my day off today - excellent work the Release Boys :beer:
I forgot to write that I needed to remove some output files (mostly netcdf files) for the recipe runs that required lots of disk space. The reason is that the recipe outputs are copied to the disk of the bot VM. Disk space there is limited to 1TB while some recipes produce 10GB - 100GB of data (even with the /preproc direcrotry removed). In some cases, there was not much left to check as some recipes do not produce plots but large nc files.
To help checking recipes, I have rerun 2 recipes successfully: recipe_wflow (all netcdf files kept) and recipe_daily_era5 (some netcdf files kept). @axel-lauer: do you want to have another look at these 2 recipes?
@axel-lauer: do you want to have another look at these 2 recipes?
Thanks @remi-kazeroni. As far as I can tell, the output of the two recipes looks OK. I updated the list above.
Closing this now since v2.5.0 is out :tada: Thanks to everyone who contributed to this!
An issue is open for every failing recipe; let's move the discussion to these. Cheers!!
@ESMValGroup/esmvaltool-developmentteam
The next release of ESMValTool and ESMValCore are approaching (see schedule). As part of the release process, a candidate release has been made for the Core (v2.5.0rc2) following the feature freeze announcement.
We have now tested all the ESMValTool recipes against the candidate release rc2 for the Core (see next post).
Please note that the feature freeze for the Tool is schedule for Monday, Feb 21. Please let us know as soon as possible if you are still working on something for this release by adding your issue or pull request to our v2.5.0 milestone: https://github.com/ESMValGroup/ESMValTool/milestone/8