Recipe testing using Core release candidate v2.8.1 rc1

[remi-kazeroni]

This issue documents the round of recipe testing performed using the Core release candidate v2.8.1rc1. The goal of this round of testing is to check how well the recipes can be run with Python3.11 and esmvalcore2.8.1rc1.

System and settings

conda/mamba

(base) mamba --version
mamba 1.4.2
conda 23.3.1

Git branches

testing_v281rc1 - see also https://github.com/ESMValGroup/ESMValTool/pull/3187

Installation and environment

$ cd ~/ESMValTool
$ mamba env create -n dev_v281rc1 -f environment.yml
$ conda activate dev_v281rc1
$ pip install --editable '.[develop]'

ESMValTool version

$ esmvaltool version
ESMValCore: 2.8.1rc1
ESMValTool: 2.9.0.dev19+g85f568252

Environment file

dev_core281rc1.txt

Script to run recipes:

See https://docs.esmvaltool.org/en/latest/utils.html#using-generate-py

Attention: @ESMValGroup/technical-lead-development-team

[remi-kazeroni]

Overview of the results

Numbers of successes and failures

The first round of recipe testing produced:

• 137 successful runs (i.e. where a index.html file was generated)
• 14 failed runs:
  • 2 Julia diagnostics (my installation does not work, I think it's just me)
  • 3 recipes could not run in 8 hours (max time on Levante)
  • 2 known issues (see our reference list of broken recipes)
  • 7 diagnostics failed, mostly due to a new version of the pandas package. This was at 1.5.3 when we released v2.8.0 and is now at 2.0.1. I will open an issue in ESMValTool.

Overview webpage and path to data

• Webpage: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.1rc1/
• Path to the runs on Levante (with preproc dirs for failed runs): /work/bd0854/b309192/recipe_testing/recipe_testing_v2p81_rc1/scripts/esmvaltool_output

Output comparison

When comparing these output against those produced with v2.8.0 using this tool, it is reported that only 10 plots (out of 9500!) differ! 🥳 I could check the 10 figures manually and did not see anything critical. Note that the version of matplotlib has not changed since the release of v2.8.0. Details below:

```bash grep -B 3 "\.png" compare_v281rc1_output.txt Reference run: /shared/esmvaltool/v2.8.0/recipe_flato13ipcc_figure_96_20230321_162658 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_flato13ipcc_figure_96_20230522_130810 Differing files: - plots/fig09-6_cor_collect/fig09-6_cor_collect/patterncor.png -- Reference run: /shared/esmvaltool/v2.8.0/recipe_ipccwg1ar6ch3_fig_3_43_20230321_152412 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_ipccwg1ar6ch3_fig_3_43_20230522_130326 Differing files: - plots/fig_3_43/cor_collect/patterncor.png -- Reference run: /shared/esmvaltool/v2.8.0/recipe_kcs_20230321_120801 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_kcs_20230522_131824 Differing files: - plots/local_resampling/resample/local_validation_2050.png - plots/local_resampling/resample/local_validation_2085.png -- Reference run: /shared/esmvaltool/v2.8.0/recipe_modes_of_variability_20230321_123029 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_modes_of_variability_20230522_154650 Differing files: - plots/weather_regime/main/SON-zg_predicted_regimes.png - plots/weather_regime/main/SON-zg_reference_regimes.png - plots/weather_regime/main/Table_zg_SON_rmse_CNRM-CM5_2020-01-01 01:00:00_2075-12-31 01:00:00_1971-01-01 01:00:00_2000-12-31 01:00:00.png -- Reference run: /shared/esmvaltool/v2.8.0/recipe_multimodel_products_20230321_124704 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_multimodel_products_20230522_134212 Differing files: - plots/anomaly_agreement/main/tas_JUN_multimodel-anomaly_2006_2099_1961_1990.png -- Reference run: /shared/esmvaltool/v2.8.0/recipe_spei_20230321_120309 Current run: /shared/esmvaltool/v2.8.1rc1/recipe_spei_20230522_141008 Differing files: - plots/diagnostic/spei/histplot.png - plots/diagnostic/spi/histplot.png ```

[remi-kazeroni]

Recipes runs that last abnormally long

Several recipes took much longer to run than with the release environment of v2.8.0 and that is reproducible on Levante. For example:

• recipe_ipccwg1ar6ch3_fig_3_19 from 1h30 to 7h30
• recipe_lauer22jclim_fig3-4_zonal and recipe_lauer22jclim_fig5_lifrac from 3 hours to more than 8 hours
• recipe_collins13ipcc from 3 hours to more than 8 hours

All runtime settings were identical between v2.8.0 and v2.8.1rc1. The @ESMValGroup/technical-lead-development-team suspects that this comes from one of our dependencies (dask, hdf5, ...). See also https://github.com/ESMValGroup/ESMValTool/pull/3187 More tests for these recipes will be done next week.

[bouweandela]

Nice work @remi-kazeroni! I've looked at recipe_modes_of_variability.yml in the past and the difficulty in comparing runs there is that algorithm does not produce exactly the same results for every run and that the order of the subplots changes between runs. Maybe this could be remedied by setting a random seed value somewhere..

[valeriupredoi]

indeed - great work @remi-kazeroni :beers: And as @bouweandela mentions, that variability recipe made me scratch my head for 2.7 too - I guess the name suggests it, maybe we should rename it to recipe_english_weather.yml

[remi-kazeroni]

Closing this as v.2.8.1 was released and further testing will be done soon for the release of v2.9.0