search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
224 stars 128 forks source link

Recipe testing for release 2.8.0 - Core release candidate rc1 #3076

Closed remi-kazeroni closed 1 year ago

remi-kazeroni commented 1 year ago

This issue documents the round of recipe testing performed using the Core release candidate v2.8.0rc1.

Release process

System and settings

conda/mamba

(base) mamba --version
mamba 1.3.1
conda 23.1.0

Git branches and state

Tue 7 Mar 16:06:43 CET 2023

(base) :~/ESMValTool 
$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

(base) :~/ESMValCore 
$ git status
On branch v2.8.x
Your branch is up to date with 'origin/v2.8.x'.

nothing to commit, working tree clean

Installation and environment

$ cd ~/ESMValTool
$ mamba env create -n tool_280rc1 -f environment.yml
$ conda activate tool_280rc1
$ pip install --editable '.[develop]'
$ cd ~/ESMValCore
$ pip install --editable '.[develop]'

Config user file

Main options: all default except search_esgf: when_missing

```yaml output_dir: ./esmvaltool_output max_parallel_tasks: 8 log_level: debug exit_on_warning: false output_file_type: png remove_preproc_dir: true compress_netcdf: false save_intermediary_cubes: false config_developer_file: null profile_diagnostic: false # Site-specific entries: DKRZ-Levante search_esgf: when_missing download_dir: /work/bd0854/DATA/ESMValTool2/download auxiliary_data_dir: /work/bd0854/DATA/ESMValTool2/AUX rootpath: CMIP6: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ CMIP5: /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ CMIP3: /work/bd0854/DATA/ESMValTool2/CMIP3 CORDEX: /work/ik1017/C3SCORDEX/data/c3s-cordex/output OBS: /work/bd0854/DATA/ESMValTool2/OBS OBS6: /work/bd0854/DATA/ESMValTool2/OBS obs4MIPs: /work/bd0854/DATA/ESMValTool2/OBS ana4mips: /work/bd0854/DATA/ESMValTool2/OBS native6: /work/bd0854/DATA/ESMValTool2/RAWOBS RAWOBS: /work/bd0854/DATA/ESMValTool2/RAWOBS drs: CMIP6: DKRZ CMIP5: DKRZ CMIP3: DKRZ CORDEX: BADC obs4MIPs: default ana4mips: default OBS: default OBS6: default native6: default ```

ESMValTool version

$ esmvaltool version
ESMValCore: 2.8.0rc1
ESMValTool: 2.8.0.dev77+gd0c0c038e

Environment file

tool_280rc1.txt

Compute resources used

On DKRZ-Levante

Note: no output comparison will be done for this round of testing. The main purpose is to identify issues with the Core and start fixing failing diagnostics. Results and overview webpage will be posted in a later post.

remi-kazeroni commented 1 year ago

Overview of the results

Numbers of successes and failures

The first round of recipe testing produced:

For comparison, we released ESMValTool 2.7.0 with 4 non-working recipes (this could have been 8 if we used a stricter policy on missing data as done for this round of testing)

Overview webpage and path to data

Note: I will now write three more posts to this issue for each category of failures (missing data, diagnostic issues, preprocessor issues) and provide more details about failures and lists of affected recipes. Please do not use this issue to comment on single recipe failures but rather open a separate issue, linking to this one. This issue is intended to provide a general overview or discuss recurring problems. More info and tagging the community soon 👍

valeriupredoi commented 1 year ago

@remi-kazeroni great work on this, bud! I see some manifestations of #3061 - so that might not be just a mocking test error; maybe @bouweandela has some time to look at, at least, anav failure :beer:

remi-kazeroni commented 1 year ago

Recipe failures due to missing data or files or data issues

Recipe Test run Problem Assigned to Related issue PR
recipe_esacci_lst link new ancestor taks issue due to malformed CMORized obs - #3104
recipe_flato13ipcc link known missing data issue @katjaweigel #2156
recipe_tebaldi21esd link new missing data issue: pr-tas, Amon, CMIP6, MPI-ESM1-2-HR, ScenarioMIP, ssp370, r* @remi-kazeroni https://github.com/ESMValGroup/ESMValTool/pull/3093
recipe_autoassess_landsurface_soilmoisture link known missing climatology files (non-public) @alistairsellar @valeriupredoi marked as broken in https://github.com/ESMValGroup/ESMValTool/issues/3103
recipe_schlund20jgr_gpp* link to one known retracted data issue @schlunma successfully tested with an extended data pool
remi-kazeroni commented 1 year ago

Recipe failures due to diagnostic errors

Recipe Test run Problem Assigned to Related issue PR
recipe_arctic_ocean link matplotlib issue - #3112
recipe_autoassess_stratosphere link matplotlib issue - #3088
recipe_bock20jgr_fig_8-10 link matplotlib issue @schlunma fixed by https://github.com/ESMValGroup/ESMValTool/pull/3087
recipe_carvalhais14nat link matplotlib issue - -
recipe_hype link numpy issue @Peter9192 #3101
recipe_monitor link matplotlib issue @sloosvel see #3085
recipe_perfmetrics_CMIP5* link to one issue with perfmetrics/main.ncl @LisaBock #3083 #3098
recipe_smpi* link to one issue with perfmetrics/main.ncl - #3098
recipe_sea_surface_salinity link matplotlib issue -@sloosvel see #3100
remi-kazeroni commented 1 year ago

Recipe failures due to preprocessor errors

Recipe Test run Problem Assigned to Related issue PR
recipe_anav13jclim link issue with weighting_landsea_fraction for OBS data @schlunma run fine with https://github.com/ESMValGroup/ESMValTool/pull/3064
recipe_autoassess_radiation_rms_cfMon_all link known CMOR check issue with CMOR checks - https://github.com/ESMValGroup/ESMValCore/issues/1238 but won't fix for v2.8?
recipe_bock20jgr_fig_1-4 link preproc issue reading OBS_HadCRUT4 - https://github.com/ESMValGroup/ESMValCore/issues/1962
recipe_check_obs link known derivation issue for ERA5 - see https://github.com/ESMValGroup/ESMValCore/issues/1388
recipe_climate_change_hotspot link new from_datasets.py issue @sloosvel #3086
recipe_collins13ipcc link new out of memory issue - #3106
recipe_meehl20sciadv link new _io.py concatenation issue @schlunma run fine with #3064
recipe_russell18jgr link new preprocessor/_supplementary_vars.py issue @schlunma run fine with https://github.com/ESMValGroup/ESMValTool/pull/3064
recipe_schlund20esd link new _io.py concatenation issue @schlunma run fine with https://github.com/ESMValGroup/ESMValTool/pull/3064
recipe_wenzel16jclim link new preprocessor/_supplementary_vars.py issue @schlunma #3084; run fine with https://github.com/ESMValGroup/ESMValTool/pull/3064
remi-kazeroni commented 1 year ago

Hi @ESMValGroup/esmvaltool-developmentteam and @ESMValGroup/esmvaltool-recipe-maintainers, the results from the first round of recipe testing for the release of ESMValTool and ESMValCore v2.8 are now available. See summary and link to the overview webpage in this https://github.com/ESMValGroup/ESMValTool/issues/3076#issuecomment-1460111541 above.

It would be great if you could take a look at the results of your favourite recipes and help fixing the failing ones. In such cases, please assign yourselves in the tables above and if possible open separate issues to discuss the problems. Feel free to comment here if you have noticed other problems with the recipe output.

There are 3 main categories of failures:

Any help will be greatly appreciated 👍 Please tag me in the PRs for recipe fixing, I can take a look and merge them into the main and release branches. Thanks in advance for your support!

valeriupredoi commented 1 year ago

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7. pin in environment.yml and make it either 2.8. or remove it completely, and to activate the rc channel :+1:

valeriupredoi commented 1 year ago

@remi-kazeroni one thing you should tell the users that will be running their recipes is to deactivate the esmvalcore 2.7. pin in environment.yml and make it either 2.8. or remove it completely, and to activate the rc channel +1

hang on, we have an issue with the Tool env picking up Core 2.8.0rc1 off the rc channel/label - am looking into this atm :+1:

valeriupredoi commented 1 year ago

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir :+1:

remi-kazeroni commented 1 year ago

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir 👍

Thanks for checking, good to know it runs fine 👍

We have started to discuss how to handle this recipe in the issue on the broken recipe policy #3065 (see this https://github.com/ESMValGroup/ESMValTool/issues/3065#issuecomment-1460526522). If the clima files cannot be publicly accessed and if that access cannot be documented, we will always have trouble testing such recipes for releases.

Remember that we have discussed a lot already to avoid these manual steps where the release manager would need to search for extra files to run particular recipes. This makes the release process too complicated otherwise. I think you have a PR that aims at simplifying the description of the release process: #3032 (I'm going to push it to the finish line soon) 👍

valeriupredoi commented 1 year ago

@remi-kazeroni my apologies - for the autoassess soilmoisture recipe, I plopped the clima files at /home/b/b382109/autoassess_files - just did a run with the new rc1 and that recipe runs well! If you want me to, I can put those files elsewhere, if you don't have access to them from me HOME dir +1

Thanks for checking, good to know it runs fine +1

We have started to discuss how to handle this recipe in the issue on the broken recipe policy #3065 (see this #3065 (comment)). If the clima files cannot be publicly accessed and if that access cannot be documented, we will always have trouble testing such recipes for releases.

Remember that we have discussed a lot already to avoid these manual steps where the release manager would need to search for extra files to run particular recipes. This makes the release process too complicated otherwise. I think you have a PR that aims at simplifying the description of the release process: #3032 (I'm going to push it to the finish line soon) +1

totally with you on that, bud! ah that #3032 - had forgotten about it :grin: BTW great work streamlining it, such a socialist/community PR :hammer_and_pick:

remi-kazeroni commented 1 year ago

Thanks everyone for your support with fixing recipes! Most failed recipes have received some attention and many of them got fixed already. That's great news for the upcoming release 👍

Up to now, it seems that 2 recipes have unfortunately not received attention:

If you happen to know their maintainers or colleagues/friends or if you have encountered similar diagnostic issues recently, you are most welcome to jump in here and help us getting the recipes fixed for the upcoming release. Otherwise these would be marked as broken in ESMValTool v2.8.

valeriupredoi commented 1 year ago

@remi-kazeroni hold on tight buddy, Carvalhais will get a solution in the next few minutes, I know exactly what's up there

valeriupredoi commented 1 year ago

Carvalhais needs to get back to the other patches - @zklaus fixed that for when we pinned Matplotlib last release, see https://github.com/ESMValGroup/ESMValTool/issues/2886#issuecomment-1292135500 - now we need to revert that, I'll PR it now

valeriupredoi commented 1 year ago

Carvalhais14 fixed here https://github.com/ESMValGroup/ESMValTool/pull/3111

TomasTorsvik commented 1 year ago

Hi, I looked at the recipe_arctic_ocean log, and apparently the cm.register_cmap function has been deprecated in matplotlib since version 3.7 (see https://matplotlib.org/stable/api/cm_api.html#matplotlib.cm.register_cmap ) and is suggested to be replaced by matplotlib.colormaps.register. Could it be a case of just replacing the deprecated function?

Unfortunately, I'm not able to run any test at the moment. The Norwegian national data storage is migrating all data to a new location, and have closed down access for some time.

remi-kazeroni commented 1 year ago

colormaps.register

Hi @TomasTorsvik, thanks very much for taking a look and your suggestion! It would be great if you or someone could make these modifications available in a branch and open a (draft) pull request with that. This would then ease the testing for developers like me with access to all necessary data at DKRZ. Note that you could also make use of our bot directly from that PR (see doc). This would allow you to request a fresh installation of ESMValTool and a test run at DKRZ directly via the PR.

remi-kazeroni commented 1 year ago

Thanks everyone for you help and support with fixing recipes! All cases have now been addressed. I'm closing this and will open tomorrow another issue for the second and last round of testing (currently ongoing) where the community will be asked to take a look at recipe output and status of the comparison with results from the previous release.

valeriupredoi commented 1 year ago

legend @remi-kazeroni :medal_military: Hopefully the next (and last) round will be a cake walk :cake: