search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
211 stars 124 forks source link

Recipe running results for v2.6.0rc3 and v2.6.0rc4 #2704

Closed sloosvel closed 2 years ago

sloosvel commented 2 years ago

@ESMValGroup/esmvaltool-developmentteam @ESMValGroup/esmvaltool-recipe-maintainers please, feel free to check the recipe running results using ESMValCore v2.6.0rc3 in the following link: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc3/

There have not been any changes with respect v2.6.0rc2 in terms of failures when running the recipes.

However the recipe comparison tool developed by @bouweandela reports that 79 recipes need to be inspected by a human due to differences in the results with respect to v2.5.0:

Summary

recipe_albedolandcover.yml: /shared/esmvaltool/v2.6.0rc3/recipe_albedolandcover_20220701_070913 recipe_climwip_brunner2019_med.yml: /shared/esmvaltool/v2.6.0rc3/recipe_climwip_brunner2019_med_20220701_070951 recipe_climwip_brunner20esd.yml: /shared/esmvaltool/v2.6.0rc3/recipe_climwip_brunner20esd_20220701_074428 recipe_climwip_test_basic.yml: /shared/esmvaltool/v2.6.0rc3/recipe_climwip_test_basic_20220701_074404 recipe_concatenate_exps.yml: /shared/esmvaltool/v2.6.0rc3/recipe_concatenate_exps_20220701_080134 recipe_correlation.yml: /shared/esmvaltool/v2.6.0rc3/recipe_correlation_20220701_080151 recipe_deangelis15nat.yml: /shared/esmvaltool/v2.6.0rc3/recipe_deangelis15nat_20220701_073236 recipe_decadal.yml: /shared/esmvaltool/v2.6.0rc3/recipe_decadal_20220701_080248 recipe_eady_growth_rate.yml: /shared/esmvaltool/v2.6.0rc3/recipe_eady_growth_rate_20220701_071105 recipe_ecs_scatter.yml: /shared/esmvaltool/v2.6.0rc3/recipe_ecs_scatter_20220701_072257 recipe_esacci_lst.yml: /shared/esmvaltool/v2.6.0rc3/recipe_esacci_lst_20220701_071326 recipe_esacci_oc.yml: /shared/esmvaltool/v2.6.0rc3/recipe_esacci_oc_20220701_073012 recipe_extract_shape.yml: /shared/esmvaltool/v2.6.0rc3/recipe_extract_shape_20220701_080152 recipe_eyring06jgr.yml: /shared/esmvaltool/v2.6.0rc3/recipe_eyring06jgr_20220701_071318 recipe_eyring13jgr_12.yml: /shared/esmvaltool/v2.6.0rc3/recipe_eyring13jgr_12_20220701_071013 recipe_hype.yml: /shared/esmvaltool/v2.6.0rc3/recipe_hype_20220701_075500 recipe_julia.yml: /shared/esmvaltool/v2.6.0rc3/recipe_julia_20220701_080222 recipe_li17natcc.yml: /shared/esmvaltool/v2.6.0rc3/recipe_li17natcc_20220701_071843 recipe_my_personal_diagnostic.yml: /shared/esmvaltool/v2.6.0rc3/recipe_my_personal_diagnostic_20220701_080034 recipe_ocean_amoc.yml: /shared/esmvaltool/v2.6.0rc3/recipe_ocean_amoc_20220701_071014 recipe_ocean_ice_extent.yml: /shared/esmvaltool/v2.6.0rc3/recipe_ocean_ice_extent_20220701_073956 recipe_ocean_quadmap.yml: /shared/esmvaltool/v2.6.0rc3/recipe_ocean_quadmap_20220701_070916 recipe_ocean_scalar_fields.yml: /shared/esmvaltool/v2.6.0rc3/recipe_ocean_scalar_fields_20220701_070912 recipe_pcrglobwb.yml: /shared/esmvaltool/v2.6.0rc3/recipe_pcrglobwb_20220701_075619 recipe_preprocessor_derive_test.yml: /shared/esmvaltool/v2.6.0rc3/recipe_preprocessor_derive_test_20220701_080318 recipe_preprocessor_test.yml: /shared/esmvaltool/v2.6.0rc3/recipe_preprocessor_test_20220701_080215 recipe_python.yml: /shared/esmvaltool/v2.6.0rc3/recipe_python_20220701_080304 recipe_r.yml: /shared/esmvaltool/v2.6.0rc3/recipe_r_20220701_080104 recipe_radiation_budget.yml: /shared/esmvaltool/v2.6.0rc3/recipe_radiation_budget_20220701_072419 recipe_seaice_drift.yml: /shared/esmvaltool/v2.6.0rc3/recipe_seaice_drift_20220701_074322 recipe_seaice_feedback.yml: /shared/esmvaltool/v2.6.0rc3/recipe_seaice_feedback_20220701_072347 recipe_snowalbedo.yml: /shared/esmvaltool/v2.6.0rc3/recipe_snowalbedo_20220701_073825 recipe_validation_CMIP6.yml: /shared/esmvaltool/v2.6.0rc3/recipe_validation_CMIP6_20220701_070916 recipe_variable_groups.yml: /shared/esmvaltool/v2.6.0rc3/recipe_variable_groups_20220701_080222 recipe_williams09climdyn_CREM.yml: /shared/esmvaltool/v2.6.0rc3/recipe_williams09climdyn_CREM_20220701_070943

So if you are a human and your recipe is needs to be inspected, please check the recipe box and report back if there are any major differences in the results.

schlunma commented 2 years ago

Great, thanks for the overview!

Could you briefly post the paths on Levante from the rc2 and rc3 results so I can run the tool to compare the outputs myself (to get more detailed output). Thanks!

fserva commented 2 years ago

Hi, results for zmnam in https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0/recipe_zmnam_20220310_185949/ and https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc3/recipe_zmnam_20220701_070930/ look the same for me. Thanks

sloosvel commented 2 years ago

@schlunma I am afraid results from rc2 are no longer available in levante because they were in my scratch, just in https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc2

For rc3, the results are here: /scratch/b/b381943/esmvaltool_output but it's the same as what's already in https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc3

schlunma commented 2 years ago

Ah, so you compared the runs to v2.5.0? Could you point me to the location where this is stored on Levante?

sloosvel commented 2 years ago

Yes, but I ran the comparison in the esmvaltool machine, using the 2.5.0 outputs that are there. I installed the tool in my home and pointed to the results in /shared

schlunma commented 2 years ago

For recipe_anav13jclim.yml, filenames change because the order of the diagnostics in the perfmetrics plot was not fixed; this is fixed in https://github.com/ESMValGroup/ESMValTool/pull/2706/.

remi-kazeroni commented 2 years ago

Ah, so you compared the runs to v2.5.0? Could you point me to the location where this is stored on Levante?

@schlunma you can find a copy of the final output for v2.5 recipes in /work/bd0854/b309192/recipe_testing/recipe_testing_v2p5_rc6/esmvaltool_output

schlunma commented 2 years ago

Thanks @remi-kazeroni :+1:

schlunma commented 2 years ago

Yesterday I started comparing the different runs for some of the recipe (I will continue tomorrow since Levante is on maintenance today). I noticed two things so far:

  1. I think the tool to compare the recipe output is too strict w.r.t. cube attributes like filename, etc. that are expected to change between runs (https://github.com/ESMValGroup/ESMValTool/issues/2708).
  2. The output of some diagnostics depends on the order in which input files are read. I couldn't figure out where exactly this can vary between runs (e.g, the metadata.yml files are sorted, so they shouldn't change), but I guess the order in which different metadata.yml files read could change. An example is the diagnostic ecs.py, which saves scalar values for each model in a 1D cube with the single dimension dataset. The order of this dataset coordinate is not fixed and can vary between runs. This is not a problem for the scientific analysis, but nevertheless leads to different results detected by the comparison tool. I see two solutions here: (1) we somehow fix the order in which input files are given to diagnostics (which can have unintentional side effects; maybe some diagnostics actually depend on the order in which files are given?), or (2) I add sorting to all affected diagnostics.
bouweandela commented 2 years ago

(2) I add sorting to all affected diagnostics.

This sounds easiest, maybe just sort the datasets alphabetically?

katjaweigel commented 2 years ago

Although marked as unchanged, some peculiar additional frames around maps appeared in recipe_li17natcc.yml: /shared/esmvaltool/v2.6.0rc3/recipe_li17natcc_20220701_071843 as well as in recipe_martin18grl.yml: /shared/esmvaltool/v2.6.0rc3/recipe_martin18grl_20220701_071025 I'll open an issue/PR for that when Levante is working again (@lukruh ).

schlunma commented 2 years ago

(2) I add sorting to all affected diagnostics.

This sounds easiest, maybe just sort the datasets alphabetically?

Yes, I think so, too. I already started implementing this, will continue tomorrow.

zklaus commented 2 years ago

@schlunma, @bouweandela, wrt sorting, as far as I can see the metadata.yml files use simple mappings (with complex keys). These don't have an order. We might try to change that to ordered mappings to increase reproducibility (and possibly obviate the need to imply a potentially different post-hoc order). I am not sure if - ? works; another possibility might be !!omap cf Example 2.26 in the yaml spec.

schlunma commented 2 years ago

Ah, so the sorting done before saving the file is not sufficient?

zklaus commented 2 years ago

No, because there metadata is an ordinary dictionary and those don't have an order. If you replace in this line

metadata = {}

with

metadata = OrderedDict()

and add the corresponding from collections import OrderedDict at the top, the yaml file should change to include something like

--- !!omap

at the top which will clue in the reader on the other side to heed the order.

schlunma commented 2 years ago

I though regular dicts are ordered since Python 3.7?

see e.g., here:

[...] regular dicts have guaranteed ordering since Python 3.7

zklaus commented 2 years ago

You are right, that escaped me. However, the yaml data model doesn't take that into account, see yaml spec Section 1.2.2. Of course, that might still be unrelated :shrug:, but if we think the order should be reflected in the yaml file, we could use an ordered dict.

sloosvel commented 2 years ago

Thank you all for taking a look, I will wait until Friday for the remaining recipes. If by then they are not taken care of, I will take a quick look. But bear in mind that I am not going to look in depth, so some issues may go unnoticed.

lukruh commented 2 years ago

Although marked as unchanged, some peculiar additional frames around maps appeared in recipe_li17natcc.yml: /shared/esmvaltool/v2.6.0rc3/recipe_li17natcc_20220701_071843 as well as in recipe_martin18grl.yml: /shared/esmvaltool/v2.6.0rc3/recipe_martin18grl_20220701_071025 I'll open an issue/PR for that when Levante is working again (@lukruh ).

This seems not directly related to the change from v2.5 to v2.6 (maybe it is intreduced by a new version of matplotlib?). However, I opened an Issue here: #2711

sloosvel commented 2 years ago

@ESMValGroup/esmvaltool-developmentteam please find below a summary of the changes for the remaining recipes.

I will split it in several comments becaue the summary is rather large.

Summary of remaining recipes

Complains of differing netcdf files, however there seems to be no issue with any on the png files that get generated. So I am going to assume that the differences are coming from some sort of attribute and that they are not a big deal:

  - work/arctic_ocean/arctic_ocean_main/arctic_ocean_ACCESS1-0_so_timmean.nc:
    attributes are different:
      value of attribute 'CDI' is different: expected 'Climate Data Interface version 2.0.5 (https://mpimet.mpg.de/cdi)' but found 'Climate Data Interface version 2.0.4 (https://mpimet.mpg.de/cdi)'
      value of attribute 'CDO' is different: expected 'Climate Data Operators version 2.0.5 (https://mpimet.mpg.de/cdo)' but found 'Climate Data Operators version 2.0.4 (https://mpimet.mpg.de/cdo)'
        data is different

Complains about the following csv differing files: plots/aa_landsurf_permafrost/autoassess_landsurf_permafrost/ACCESS-CM2_vs_E3SM-1-0/land_surface_permafrost/ACCESS-CM2/metrics.csv and plots/aa_landsurf_permafrost/autoassess_landsurf_permafrost/ACCESS-CM2_vs_E3SM-1-0/land_surface_permafrost/E3SM-1-0/metrics.csv

Checked and the numerical results are not quite identical due to precision, for instance:

    -pr: MPI-ESM-LR vs NorESM1-M: 3.574050222311568e-06
    -pr: MPI-ESM-LR vs GPCP-SG: 5.293138857953477e-06
    +pr: MPI-ESM-LR vs NorESM1-M: 3.5740502221844485e-06
    +pr: MPI-ESM-LR vs GPCP-SG: 5.293138857050933e-06

No comparison, because the diagnostic fails in v2.5.0 . I guess because this is one of the metrics that needs to run in jasmin.

Complains about cvs files, same issue as in recipe_autoassess_landsurf_permafrost

Complains about cvs files, same issue as in recipe_autoassess_landsurf_permafrost

No differences in plots nor work files, just complains about the presence of extra files that I guess were deleted from the 2.5 outputs.

Complains about some differing netcdf files, but no issues with any of the plots files. There are complaints both about the data being different and some attributes. I can't comment on the data being different.

Complains about differing results in both netcdf and a plot about patterncor:

Differing files:
  - plots/fig_7/cor_collect/patterncor.png
  - work/collect/fig_6_RMSD/ta850-global_to_swcre-global_RMSD.nc
  - work/fig_7/cor_collect/pattern_cor.nc
  - work/pr_cor/pattern_cor/pattern_cor.nc
  - work/psl_cor/pattern_cor/pattern_cor.nc
  - work/rlut_cor/pattern_cor/pattern_cor.nc
  - work/swcre_cor/pattern_cor/pattern_cor.nc
  - work/tas_cor/pattern_cor/pattern_cor.nc

Can't really comment on these.

Complains about differing plots/capacity_factor/main/capacity_factor_CMCC-CMS_1980-2005.png plot, checked and it seems like the size of the plot is different but the results are identical.

Complains about differing netcdf files, but no issues with any of the plots files. Differences are mostly due to differing attributes but there is one file where data is different: CMIP5_NorESM1-M_Lmon_historical_r1i1p1_cSoil_1982-2005.nc' but found '/mnt/lustre02/work/bd0854/b309192/recipe_testing_v2p5_rc6/esmvaltool_output/recipe_carvalhais14nat_20220310_191019/preproc/diag_land_carbon_turnover/ctotal_derive_input_cSoil/CMIP5_NorESM1-M_Lmon_historical_r1i1p1_cSoil_1982-2005.nc' data is different data is different data is different 

Differing files:
  - plots/calculate_weights_climwip/climwip/performance_overall_mean.png
  - plots/calculate_weights_climwip/climwip/performance_pr_CLIM.png
  - plots/calculate_weights_climwip/climwip/performance_sigma_calibration.png
  - plots/calculate_weights_climwip/climwip/performance_tas_CLIM.png
  - plots/calculate_weights_climwip/climwip/weights.png
  - plots/weighted_temperature_graph/weighted_temperature_graph/temperature_anomaly_graph.png
  - plots/weighted_temperature_map/weighted_temperature_map/temperature_change_difference_map.png
  - plots/weighted_temperature_map/weighted_temperature_map/temperature_change_weighted_map.png
  - work/calculate_weights_climwip/climwip/performance_overall_mean.nc
  - work/calculate_weights_climwip/climwip/performance_pr_CLIM.nc
  - work/calculate_weights_climwip/climwip/performance_sigma_calibration.nc
  - work/calculate_weights_climwip/climwip/performance_tas_CLIM.nc
  - work/calculate_weights_climwip/climwip/performance_tas_TREND.nc
  - work/calculate_weights_climwip/climwip/weights.nc
  - work/weighted_temperature_map/weighted_temperature_map/temperature_change_difference_map.nc
  - work/weighted_temperature_map/weighted_temperature_map/temperature_change_weighted_map.nc

However, it's the only climwip recipe that gives issues, and it happens to be the test one not the "real" ones. So I am going to assume whatever is the issue, is not a big deal.

Complains about differing netcdf files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Complains about differing netcdf files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Complains because the run for rc3 did not finish because of the job submission. But the output comparison for rc2 with respect to 2.5 gives identical results

Complains about plots/combine_indices/main/Nino3.4_tos_Dec-Feb_running-mean__1950-2005.png but the results look identical to me

Complains about a plot, probably related to #2671

Complains about netcdf files and some namelists. However, whatever may be the issue it can't be related to the core because cvdp is an external package.

Complains about extra files present between runs that were probably deleted form the 2.5 outputs.

Complains about a plot but it looks identical to me

Complains about extra files present between runs that were probably deleted form the 2.5 outputs.

Complains about plots, but they look identical to me. This is happening in all magic recipes developed by the BSC.

Complains about extra files present between runs that were probably deleted form the 2.5 outputs.

Another magic recipe that complains about the plots but that, to my inexpert eye, look the same.

Complains about differing netcdf files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Differing files:
  - plots/hyint/main/dsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/dsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/dsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/dsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/dsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/dsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/dsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/dsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/hyint_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/hyint_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/hyint_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/hyint_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/hyint_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/hyint_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/hyint_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/hyint_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/int_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/int_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/int_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/int_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/int_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/int_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/int_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/int_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/multiindex_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_timeseries.png
  - plots/hyint/main/multiindex_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_trend_summary.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_timeseries.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_trend_summary.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/multiindex_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_timeseries.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_trend_summary.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/multiindex_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_timeseries.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL__multiregion_trend_summary.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/multiindex_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/multiindex_multimodel_1980-2020_ALL__GL_timeseries.png
  - plots/hyint/main/multiindex_multimodel_1980-2020_ALL__GL_trend_summary.png
  - plots/hyint/main/pa_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/pa_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/pa_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/pa_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/pa_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/pa_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/pa_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/pa_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/r95_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/r95_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/r95_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/r95_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/r95_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/r95_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/r95_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/r95_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/wsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/wsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/wsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/wsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/wsl_norm_ACCESS1-0_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/wsl_norm_CCSM4_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/wsl_norm_IPSL-CM5A-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_comp_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_AF_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_comp_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EA_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_comp_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_EU_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_comp_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_GL_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_comp_map.png
  - plots/hyint/main/wsl_norm_MPI-ESM-MR_historical-rcp85_r1i1p1_1980-2020_ALL_myearmean_SA_map.png

I cannot really comment on that. I doubt it's an issue of the core because the only thing the preproc does is using mask_landsea

sloosvel commented 2 years ago

Complains about differing plots and netcdf. I doubt it's an issue of the core because the only thing the preproc does is using mask_landsea

Complains about differing netcdf and plots. Differing files:

  - plots/landcover/landcover/area_baresoilFrac.png
  - plots/landcover/landcover/area_cropFrac.png
  - plots/landcover/landcover/area_grassFrac.png
  - plots/landcover/landcover/area_shrubFrac.png
  - plots/landcover/landcover/area_treeFrac.png
  - plots/landcover/landcover/bias_baresoilFrac.png
  - plots/landcover/landcover/bias_cropFrac.png
  - plots/landcover/landcover/bias_grassFrac.png
  - plots/landcover/landcover/bias_shrubFrac.png
  - plots/landcover/landcover/bias_treeFrac.png
  - plots/landcover/landcover/frac_baresoilFrac.png
  - plots/landcover/landcover/frac_cropFrac.png
  - plots/landcover/landcover/frac_grassFrac.png
  - plots/landcover/landcover/frac_shrubFrac.png
  - plots/landcover/landcover/frac_treeFrac.png
  - work/landcover/landcover/postproc_cropFrac.nc
  - work/landcover/landcover/postproc_grassFrac.nc
  - work/landcover/landcover/postproc_shrubFrac.nc
  - work/landcover/landcover/postproc_treeFrac.nc

Can't comment on that. I doubt it's an issue with the core because the only thing the preproc does is using regrid.

Complains about differing netcdf files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Complains about differing netcdf files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Complains about differing mat files, but no issues with any of the plots files. I am going to assume the difference are due to attributes, rather than issues with the data.

Complains about the presence of extra files between runs that probably were deleted from th 2.5.0 outputs

Another magic recipe that reports differences in plots but results look identical

Differences in the following plots:

Differing files:
  - plots/lineplots_anncyc_gl/lineplot_gl_anncyc/lineplot_xch4_2003-2014.png
  - plots/lineplots_anncyc_nh/lineplot_nh_anncyc/lineplot_xch4_2003-2014.png
  - plots/lineplots_anncyc_sh/lineplot_sh_anncyc/lineplot_xch4_2003-2014.png
  - plots/lineplots_gl_gr/lineplot_gl_growthrate/lineplot_xch4_2003-2014.png
  - plots/lineplots_gl_monmean/lineplot_gl_monmean/lineplot_xch4_2003-2014.png
  - plots/lineplots_nh_gr/lineplot_nh_growthrate/lineplot_xch4_2003-2014.png
  - plots/lineplots_nh_monmean/lineplot_nh_monmean/lineplot_xch4_2003-2014.png
  - plots/lineplots_sh_gr/lineplot_sh_growthrate/lineplot_xch4_2003-2014.png
  - plots/lineplots_sh_monmean/lineplot_sh_monmean/lineplot_xch4_2003-2014.png

I cannot comment on that

Another magic recipe complaining about differing results even though the plots look the same to me.

Complains about a missing file that then reappears as an extra file with a different name:

Missing files:
  - plots/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/diag_CMIP5_CanESM2_fx_historical_r0i0p0_areacello_default_diag_surface_maps_vs_OBS_map_0.png
Extra files:
  - plots/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/diag_CMIP5_CanESM2_fx_historical_r0i0p0_areacello_default_diag_surface_maps_vs_OBS_2001_2004_map_0.png

Differences on some plots due to the recently updated volume_statistics. We already detected that there were some rounding differences that affect the values in the last decimals. Nevertheless the patterns and the axes for the plots look identical between runs.

Differing netcdf and epsi files. However there are not complaints about differing plots, so it's probably an issue with metadata.

Differing plots that I cannot comment on:

Differing files:
  - plots/capacity_factor/main/capacity_factor_ACCESS1-0_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_ACCESS1-3_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_CMCC-CM_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_CNRM-CM5_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_CSIRO-Mk3-6-0_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_CanESM2_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_ERA-Interim_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_GFDL-ESM2G_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_IPSL-CM5A-MR_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_MIROC5_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_MPI-ESM-MR_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_MRI-CGCM3_1980-2005_DJF.png
  - plots/capacity_factor/main/capacity_factor_NorESM1-M_1980-2005_DJF.png

Differing plot and netcdf for a certain dataset that I cannot comment on:

- recipe_rainfarm

Complains about differing netcdf files, but no issues with any plot so I am going to assume it must be due to some attribute

Differing files:
  - plots/quantilebias/main/CanESM2_historical_r1i1p1_1979-2005_pr_75qb.png
  - work/quantilebias/main/CanESM2_historical_r1i1p1_1979-2005_pr_75qb.nc

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Complains about differing xlsx files. I guess it's the same issue about differences being reported in cvs files

Complains about the presence of extra files between runs that were probably deleted from the v2.5 runs

sloosvel commented 2 years ago

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Differences in netcdf and plots:

Differing files:
  - plots/toymodel/main/synthetic_CMIP5_bcc-csm1-1_Amon_rcp45_r1i1p1_psl_2051-2060.jpg
  - work/toymodel/main/synthetic_CMIP5_bcc-csm1-1_Amon_rcp45_r1i1p1_psl_2051-2060.nc

Can't really comment on that

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Complains about differing netcdf files but no issues in plots so I am going to assume that they are due to some attributes

Complains about differing netcdf, pdf and epsi files but no issues in plots so I am going to assume that they are due to some attributes

sloosvel commented 2 years ago

So as an overall conclusion, there are less than 10 recipes that truly seem to have an issue going on with the numerical results of the run. However, I think it would be very obvious if it was something in the core breaking because it would be affecting many recipes (especially the preprocessor testing ones).

@ESMValGroup/esmvaltool-coreteam I would say the core is ready to be officially released, but if any of you strongly oposes let me know.

zklaus commented 2 years ago

@sloosvel, don't forget to check the tests. You might want to pickup ESMValGroup/ESMValCore#1656.

bouweandela commented 2 years ago

@sloosvel There has been a lot of discussion about backward incompatible changes and deprecating stuff recently. One of the main conclusions from that is that we need to clearly list deprecated and removed features in the changelog and provide a detailed upgrade guide in the same place. I see that there is one deprecated feature (and no removed features, is that correct?) listed in the changelog, but right now it is somewhere near the bottom of the list of changes and it misses instructions on how to upgrade. I think it is important that this is addressed before the release. @schlunma Since you contributed the deprecated feature, could you please take care of adding an upgrade guide for this to the changelog and move it so it is listed right under the highlights section?

sloosvel commented 2 years ago

Ok I will open an issue in the core. But this release is already very overdue.

bouweandela commented 2 years ago

That must have been a lot of work, reviewing all those recipe runs @sloosvel, thanks for taking care!

  • recipe_arctic_ocean:

Complains of differing netcdf files, however there seems to be no issue with any on the png files that get generated. So I am going to assume that the differences are coming from some sort of attribute and that they are not a big deal.

If you run the comparison with the --verbose flag, it will tell you where the difference is (it supports attributes, coordinates, and data)

  • recipe_autoassess_radiation_rms_Amon_all.yml

Complains about cvs files, but results look identical. I guess the comparison tool is not handling very well these types of files.

It should support .csv files (see here) and if it does not than that would be a bug in the comparison tool. To see which lines are different, you can run it with the --verbose flag.

  • recipe_ocean_Landschuetzer:

Complains about a missing file that then reappears as an extra file with a different name:

Missing files:

  • plots/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/diag_CMIP5_CanESM2_fx_historical_r0i0p0_areacello_default_diag_surface_maps_vs_OBS_map_0.png Extra files:
  • plots/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/diag_CMIP5_CanESM2_fx_historical_r0i0p0_areacello_default_diag_surface_maps_vs_OBS_2001_2004_map_0.png

It might be good to check this in a bit more detail: a timerange appeared in the name of a file that is time independent and you did quite a bit of work on timeranges for this release, so I think this change could indicate a newly introduced bug in the ESMValCore.

  • recipes ocean_bgc, ocean_example, ocean_multimap

Differences on some plots due to the recently updated volume_statistics. We already detected that there were some rounding differences that affect the values in the last decimals. Nevertheless the patterns and the axes for the plots look identical between runs.

If you run the compare tool with the --verbose option, it will tell you whether the results are numerically close or quite far apart.

sloosvel commented 2 years ago

Thanks that should be helpful, I was not aware of the --verbose option.

schlunma commented 2 years ago

@schlunma Since you contributed the deprecated feature, could you please take care of adding an upgrade guide for this to the changelog and move it so it is listed right under the highlights section?

See https://github.com/ESMValGroup/ESMValCore/pull/1658

sloosvel commented 2 years ago

Running with verbose shows a lot of netcdf files where the data is different but I don't understand why this does not affect the plots in most cases. I don't think I can trace back every single source of difference.

sloosvel commented 2 years ago

It might be good to check this in a bit more detail: a timerange appeared in the name of a file that is time independent and you did quite a bit of work on timeranges for this release, so I think this change could indicate a newly introduced bug in the ESMValCore.

The work on the timeranges was mostly for the past release though. I will double check anyway.

sloosvel commented 2 years ago

If you run the compare tool with the --verbose option, it will tell you whether the results are numerically close or quite far apart.

Not on the plots though

bouweandela commented 2 years ago

Running with verbose shows a lot of netcdf files where the data is different but I don't understand why this does not affect the plots in most cases. I don't think I can trace back every single source of difference.

That is exactly why we need help from the @ESMValGroup/esmvaltool-developmentteam with making a release. Please give @sloosvel a hand with this release everyone. She has provided us with a great overview of recipes that potentially have issues. The results from the current run are available here and from the previous version here. We have a small tool available to drill down into what has changed, the documentation is available here (don't forget to use the --verbose flag if you want all the details).

bouweandela commented 2 years ago

If you run the compare tool with the --verbose option, it will tell you whether the results are numerically close or quite far apart.

Not on the plots though

@sloosvel Do you have ideas for how this could be implemented?

sloosvel commented 2 years ago

@sloosvel Do you have ideas for how this could be implemented?

As of right now, no. But again taking a look is quite clear in this case that the differences are not huge. We already discussed the source of differences in https://github.com/ESMValGroup/ESMValCore/pull/1545.

sloosvel commented 2 years ago

That is exactly why we need help from the https://github.com/orgs/ESMValGroup/teams/esmvaltool-developmentteam with making a release. Please give @sloosvel a hand with this release everyone. She has provided us with a great overview of recipes that potentially have issues. The results from the current run are available here and from the previous version here. We have a small tool available to drill down into what has changed, the documentation is available here (don't forget to use the --verbose flag if you want all the details).

I know but at some point there has to be a compromise, and it's clear that a lot of people that contributed in the past are no longer active.

I will run more comparisons between versions and also check if maybe the precision that is expected for the comparison for the netcdfs is too high to have a big impact on the plots. And if after that the missing recipes have not been claimed I think they will have to go into the release as they are, since they have probably been unmaintained for quite some time.

valeriupredoi commented 2 years ago

so here's a good point @sloosvel is raising - I'd add a level of maintenance per recipe: colour-coded: green - well maintained, yellow so-so, red - abandonware, so people know what to expect when running them. Also, if plots look the same, then I wouldn't worry too much. Cheers for all the heavy-lifting, Saskia! :beer:

schlunma commented 2 years ago

An example I found where the netcdf files differed and plots didn't were netcdf files that inlcude a dataset coordinate whose order can vary between runs. Thus, the file looks different between two runs, but basically describes the same.

sloosvel commented 2 years ago

I would also take into account that v2.5 was ran in a different machine than v2.6

valeriupredoi commented 2 years ago

I would also take into account that v2.5 was ran in a different machine than v2.6

are we actually seeing machine precision deltas? Would think those would be swamped by numerical library differences :desktop_computer: :snake:

katjaweigel commented 2 years ago

@sloosvel thanks for all the tests! I'll have a look at recipe_pv_capacity_factor now, I overlooked that in the first list. recipe_martin18grl.yml I didn't mark ok, because there are issues with the maps, some new ones, probably not really related to the release but some newer python libraries on Levante which brought some issue to the surface. I did not find any new or unusual plots so far @lukruh made a PR for this recipe_martin18grl.yml as well as recipe_li17natcc.yml, which has the same issue at github.com/ESMValGroup/ESMValTool/pull/2712

katjaweigel commented 2 years ago

The plots from recipe_pv_capacity_factor are ok, the change seems to be the font of the plot titles (new R version, maybe), looks similar for recipe_capacity_factor.yml.

sloosvel commented 2 years ago

Thanks for checking @katjaweigel !! I think many R recipes (the magic ones at least) have a similar issue in which results look similar but the tool says otherwise, so it must be due to small differences like fontsize and such.

I also took the time to do some comparisons using the outputs from 2.4.0 compared to 2.5.0, and for the following recipes, the tool reports the same differences in netcdf files as the comparison of 2.5.0 with 2.6.0:

So I am going to tick these ones out, as well because if the differences in the netcdf persist across releases but the plots seem to be fine, it can't be due to breaking changes.

sloosvel commented 2 years ago

You can find the results for release candidate 4 of ESMValCore here: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.6.0rc4/

Nothing seems to have changed with respect rc3, the comparison with 2.5.0 gives the same results. If there are no further comments, I will be doing the final release of ESMValCore tomorrow.

remi-kazeroni commented 2 years ago

Thanks for all your work on the release @sloosvel including running all the recipes and comparing the output! For the sake of completeness, you could run the comparison tool between rc3 and rc4 outputs directly to check that nothing has changed. But if you have not witnessed any changes when comparing to v2.5.0 results, I think you can proceed with the final release 👍

Do I understand correctly that the release of the Core will be soon followed by the one of the Tool? In that case, shall we close milestone v.2.6.0 and transfer open issues/PRs to milestone v2.7.0?

sloosvel commented 2 years ago

you could run the comparison tool between rc3 and rc4 outputs directly to check that nothing has changed.

Yes of course, here are the results. As you can see it's the same recipes that keep reporting the same issues. So I really think that for these selected recipes, the comparison tool is probably picking up differences that keep changing between runs regardless if they have a big impact on the results or not. Also the miles_* recipes appear now and did not before because for this run I updated the akima library (see #2695 ) just to make the recipes finish to be able to create the index page. But other than that I would say nothing has broken between runs

The following recipe runs need to be inspected by a human:
recipe_anav13jclim.yml: /shared/esmvaltool/v2.6.0rc4/recipe_anav13jclim_20220712_095747
recipe_arctic_ocean.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_arctic_ocean_20220712_094030
recipe_autoassess_landsurface_permafrost.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_autoassess_landsurface_permafrost_20220712_101605
recipe_autoassess_radiation_rms_Amon_all.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_autoassess_radiation_rms_Amon_all_20220712_093614
recipe_autoassess_radiation_rms_Amon_obs.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_autoassess_radiation_rms_Amon_obs_20220712_100232
recipe_autoassess_stratosphere.yml: /shared/esmvaltool/v2.6.0rc4/recipe_autoassess_stratosphere_20220712_101605
recipe_bock20jgr_fig_1-4.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_bock20jgr_fig_1-4_20220714_072051
recipe_bock20jgr_fig_6-7.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_bock20jgr_fig_6-7_20220713_102941
recipe_bock20jgr_fig_8-10.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_bock20jgr_fig_8-10_20220712_102546
recipe_bock20jgr_fig_8-10.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_bock20jgr_fig_8-10_20220712_132454
recipe_capacity_factor.yml: /shared/esmvaltool/v2.6.0rc4/recipe_capacity_factor_20220712_094030
recipe_carvalhais14nat.yml: /shared/esmvaltool/v2.6.0rc4/recipe_carvalhais14nat_20220712_094544
recipe_climate_change_hotspot.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_climate_change_hotspot_20220712_125737
recipe_climwip_test_performance_sigma.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_climwip_test_performance_sigma_20220712_100159
recipe_clouds_bias.yml: /shared/esmvaltool/v2.6.0rc4/recipe_clouds_bias_20220712_094030
recipe_clouds_ipcc.yml: /shared/esmvaltool/v2.6.0rc4/recipe_clouds_ipcc_20220712_093614
recipe_cmug_h2o.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_cmug_h2o_20220712_094030
recipe_combined_indices.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_combined_indices_20220712_093855
recipe_consecdrydays.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_consecdrydays_20220712_094544
recipe_cox18nature.yml: /shared/esmvaltool/v2.6.0rc4/recipe_cox18nature_20220712_094647
recipe_cvdp.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_cvdp_20220712_101636
recipe_daily_era5.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_daily_era5_20220713_103522
recipe_deangelis15nat_fig1_fast.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_deangelis15nat_fig1_fast_20220712_102931
recipe_diurnal_temperature_index.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_diurnal_temperature_index_20220712_094239
recipe_ecs.yml: /shared/esmvaltool/v2.6.0rc4/recipe_ecs_20220712_094441
recipe_ecs_constraints.yml: /shared/esmvaltool/v2.6.0rc4/recipe_ecs_constraints_20220712_093542
recipe_era5-land.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_era5-land_20220712_101708
recipe_extreme_index.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_extreme_index_20220712_094410
recipe_gier2020bg.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_gier2020bg_20220712_100159
recipe_globwat.yml: /shared/esmvaltool/v2.6.0rc4/recipe_globwat_20220712_101739
recipe_heatwaves_coldwaves.yml: /shared/esmvaltool/v2.6.0rc4/recipe_heatwaves_coldwaves_20220712_095408
recipe_hydro_forcing.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_hydro_forcing_20220712_102230
recipe_hyint.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_hyint_20220712_094513
recipe_hyint_extreme_events.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_hyint_extreme_events_20220712_094956
recipe_kcs.yml: /shared/esmvaltool/v2.6.0rc4/recipe_kcs_20220712_094030
recipe_landcover.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_landcover_20220712_100024
recipe_lauer13jclim.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_lauer13jclim_20220712_094442
recipe_lisflood.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_lisflood_20220712_102052
recipe_marrmot.yml: /shared/esmvaltool/v2.6.0rc4/recipe_marrmot_20220712_102053
recipe_martin18grl.yml: /shared/esmvaltool/v2.6.0rc4/recipe_martin18grl_20220712_094238
recipe_meehl20sciadv.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_meehl20sciadv_20220712_094308
recipe_miles_block.yml: /shared/esmvaltool/v2.6.0rc4/recipe_miles_block_20220712_094239
recipe_miles_eof.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_miles_eof_20220712_132749
recipe_miles_regimes.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_miles_regimes_20220712_093614
recipe_modes_of_variability.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_modes_of_variability_20220712_100232
recipe_monitor.yml: /shared/esmvaltool/v2.6.0rc4/recipe_monitor_20220712_103106
recipe_monitor_with_refs.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_monitor_with_refs_20220712_103002
recipe_mpqb_xch4.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_mpqb_xch4_20220712_102546
recipe_multimodel_products.yml: /shared/esmvaltool/v2.6.0rc4/recipe_multimodel_products_20220712_093855
recipe_ncl.yml: /shared/esmvaltool/v2.6.0rc4/recipe_ncl_20220712_102723
recipe_ocean_Landschuetzer2016.yml: /shared/esmvaltool/v2.6.0rc4/recipe_ocean_Landschuetzer2016_20220712_101502
recipe_ocean_bgc.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_ocean_bgc_20220712_100539
recipe_ocean_example.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_ocean_example_20220712_094718
recipe_ocean_multimap.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_ocean_multimap_20220712_094204
recipe_perfmetrics_CMIP5.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_perfmetrics_CMIP5_20220712_095440
recipe_perfmetrics_CMIP5_4cds.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_perfmetrics_CMIP5_4cds_20220712_093614
recipe_psyplot.yml: /shared/esmvaltool/v2.6.0rc4/recipe_psyplot_20220712_100609
recipe_pv_capacity_factor.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_pv_capacity_factor_20220712_094204
recipe_quantilebias.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_quantilebias_20220712_100056
recipe_rainfarm.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_rainfarm_20220712_094339
recipe_runoff_et.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_runoff_et_20220712_094030
recipe_russell18jgr.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_russell18jgr_20220712_101503
recipe_schlund20esd.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_schlund20esd_20220712_133227
recipe_schlund20jgr_gpp_abs_rcp85.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_schlund20jgr_gpp_abs_rcp85_20220712_102303
recipe_schlund20jgr_gpp_change_1pct.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_schlund20jgr_gpp_change_1pct_20220712_102335
recipe_schlund20jgr_gpp_change_rcp85.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_schlund20jgr_gpp_change_rcp85_20220712_102408
recipe_sea_surface_salinity.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_sea_surface_salinity_20220712_101503
recipe_seaice.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_seaice_20220712_101224
recipe_shapeselect.yml: /shared/esmvaltool/v2.6.0rc4/recipe_shapeselect_20220712_093647
recipe_smpi.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_smpi_20220713_134202
recipe_smpi_4cds.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_smpi_4cds_20220713_112247
recipe_spei.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_spei_20220712_094442
recipe_tcr.yml: /shared/esmvaltool/v2.6.0rc4/recipe_tcr_20220712_094412
recipe_thermodyn_diagtool.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_thermodyn_diagtool_20220712_094926
recipe_toymodel.yml:    /shared/esmvaltool/v2.6.0rc4/recipe_toymodel_20220712_095237
recipe_validation.yml:  /shared/esmvaltool/v2.6.0rc4/recipe_validation_20220712_094853
recipe_wenzel14jgr.yml: /shared/esmvaltool/v2.6.0rc4/recipe_wenzel14jgr_20220712_130053
recipe_wenzel16jclim.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_wenzel16jclim_20220712_095337
recipe_wenzel16nat.yml: /shared/esmvaltool/v2.6.0rc4/recipe_wenzel16nat_20220712_130248
recipe_wflow.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_wflow_20220712_102158
recipe_zmnam.yml:   /shared/esmvaltool/v2.6.0rc4/recipe_zmnam_20220712_094133

Action required: 81 out of 117 recipe runs need to be inspected by a human.

Do I understand correctly that the release of the Core will be soon followed by the one of the Tool? In that case, shall we close milestone v.2.6.0 and transfer open issues/PRs to milestone v2.7.0?

Ideally I would like to release the tool next week. I think it's fair to give time to include some corrections that @katjaweigel detected for her recipes (#2711) and then I need to update some documentation myself. For issues were more work is required, I am afraid some of them will have to be moved to the next milestone but I will check each one of them individually.

earnone commented 2 years ago

@sloosvel sorry for the last minute check on hyint and other recipes from us. There are only changes in the fonts, plots are identical a part from that. I am updating the list at the beginning of the issue. So hyint, hyint_extremes and quantilebias are fine, just font changes due to I guess a different version of R. I guess all R recipes will be affected in the same way. Rainfarm is fine too, the files are different because they are stocastically generated but the overall results look ok. Miles will require some more work.

sloosvel commented 2 years ago

Thank you all, version 2.6 is now available. Hopefully this first run of the comparison tool will help subsequent releases.