Closed valeriupredoi closed 1 year ago
OK folks CI tests passing GA: noice and Circle: smoothly with the new Core RC2 :parasol_on_ground:
Here is the list of recipes we must run with this guy, I will fire up the auto testing on DKRZ/Levante (as soon as I am allowed to get on to that HPC and run shtuffs), so you shouldn't be asked to run your favourite recipe just - unless you want to help out (which would be nice, but not required) and unless I specifically ask you (nicely) to :beer:
recipe_albedolandcover.yml
recipe_anav13jclim.yml
recipe_arctic_ocean.yml
recipe_autoassess_landsurface_permafrost.yml
recipe_autoassess_landsurface_soilmoisture.yml
needs https://github.com/ESMValGroup/ESMValTool/pull/2838recipe_autoassess_landsurface_surfrad.yml
needs https://github.com/ESMValGroup/ESMValTool/pull/2838recipe_autoassess_radiation_rms_Amon_all.yml
recipe_autoassess_radiation_rms_Amon_obs.yml
recipe_autoassess_radiation_rms_cfMon_all.yml
https://github.com/ESMValGroup/ESMValCore/issues/1238recipe_autoassess_stratosphere.yml
recipe_capacity_factor.yml
recipe_carvalhais14nat.yml
recipe_climate_change_hotspot.yml
see comment here and followup comments; hefty bugger, but runs fine with --skip-nonexistent
recipe_climwip_brunner2019_med.yml
recipe_climwip_brunner20esd.yml
recipe_climwip_test_basic.yml
recipe_climwip_test_performance_sigma.yml
issue raised at https://github.com/ESMValGroup/ESMValTool/issues/2847 with fix in https://github.com/ESMValGroup/ESMValTool/pull/2848recipe_clouds_bias.yml
recipe_clouds_ipcc.yml
recipe_cmug_h2o.yml
recipe_collins13ipcc.yml
recipe_combined_indices.yml
recipe_consecdrydays.yml
recipe_cox18nature.yml
recipe_cvdp.yml
recipe_deangelis15nat.yml
recipe_diurnal_temperature_index.yml
recipe_eady_growth_rate.yml
recipe_ecs_constraints.yml
recipe_ecs_scatter.yml
recipe_ecs.yml
recipe_ensclus.yml
recipe_esacci_lst.yml
recipe_esacci_oc.yml
recipe_extreme_events.yml
recipe_extreme_index.yml
recipe_eyring06jgr.yml
recipe_eyring13jgr_12.yml
recipe_flato13ipcc.yml
recipe_gier2020bg.yml
recipe_heatwaves_coldwaves.yml
recipe_hyint_extreme_events.yml
recipe_hyint.yml
recipe_impact.yml
recipe_kcs.yml
recipe_landcover.yml
recipe_lauer13jclim.yml
recipe_li17natcc.yml
recipe_martin18grl.yml
recipe_meehl20sciadv.yml
recipe_miles_block.yml
recipe_miles_eof.yml
recipe_miles_regimes.yml
recipe_modes_of_variability.yml
recipe_multimodel_products.yml
recipe_ocean_amoc.yml
recipe_ocean_bgc.yml
recipe_ocean_example.yml
recipe_ocean_ice_extent.yml
recipe_ocean_Landschuetzer2016.yml
recipe_ocean_multimap.yml
recipe_ocean_quadmap.yml
recipe_ocean_scalar_fields.yml
recipe_perfmetrics_CMIP5_4cds.yml
recipe_perfmetrics_CMIP5.yml
recipe_perfmetrics_land_CMIP5.yml
recipe_psyplot.yml
recipe_pv_capacity_factor.yml
recipe_quantilebias.yml
recipe_radiation_budget.yml
recipe_rainfarm.yml
recipe_runoff_et.yml
recipe_russell18jgr.yml
recipe_schlund20esd.yml
recipe_seaice_drift.yml
recipe_seaice_feedback.yml
recipe_seaice.yml
recipe_sea_surface_salinity.yml
recipe_shapeselect.yml
recipe_smpi_4cds.yml
possibly affected by https://github.com/ESMValGroup/ESMValTool/issues/2842 - no, the followup run on Levante from @remi-kazeroni showed all is fine and dandy (plot looks the same too)recipe_smpi.yml
recipe_snowalbedo.yml
recipe_spei.yml
recipe_tcr.yml
recipe_tebaldi21esd.yml
recipe_thermodyn_diagtool.yml
recipe_toymodel.yml
recipe_validation_CMIP6.yml
recipe_validation.yml
recipe_wenzel14jgr.yml
recipe_wenzel16jclim.yml
recipe_wenzel16nat.yml
recipe_williams09climdyn_CREM.yml
recipe_zmnam.yml
recipe_bock20jgr_fig_1-4.yml
recipe_bock20jgr_fig_6-7.yml
recipe_bock20jgr_fig_8-10.yml
recipe_daily_era5.yml
recipe_era5-land.yml
recipe_check_obs.yml
(problem with ERA5 data already spotted in previous releases, see https://github.com/ESMValGroup/ESMValCore/issues/1388)recipe_concatenate_exps.yml
recipe_correlation.yml
recipe_decadal.yml
recipe_extract_shape.yml
recipe_julia.yml
recipe_my_personal_diagnostic.yml
recipe_ncl.yml
recipe_preprocessor_derive_test.yml
recipe_preprocessor_test.yml
recipe_python.yml
recipe_r.yml
recipe_variable_groups.yml
recipe_globwat.yml
recipe_hydro_forcing.yml
recipe_hype.yml
recipe_lisflood.yml
recipe_marrmot.yml
recipe_pcrglobwb.yml
recipe_wflow.yml
recipe_ipccwg1ar6ch3_atmosphere.yml
recipe_monitor_with_refs.yml
recipe_monitor.yml
recipe_mpqb_xch4.yml
recipe_schlund20jgr_gpp_abs_rcp85.yml
plodding along - high mem requirement https://github.com/ESMValGroup/ESMValTool/issues/2845recipe_schlund20jgr_gpp_change_1pct.yml
recipe_schlund20jgr_gpp_change_rcp85.yml
plodding along - high mem requirement https://github.com/ESMValGroup/ESMValTool/issues/2845recipe_deangelis15nat_fig1_fast.yml
@ESMValGroup/esmvaltool-developmentteam I would be etrnally grateful if you could test 1-2 or more recipes of your choice and faith from the list above, please - I will be running the ones that don't get much attention from you myself, but it'd be very cool if I didn't have to run the whole list, you know :grin: For testing, please use the ESMValCore rc2 branch https://github.com/ESMValGroup/ESMValTool/tree/core_v270rc2 like this:
cd ~/ESMValTool # or alternatively if you don't have it, clone it first with git clone https://github.com/ESMValGroup/ESMValTool.git
git fetch -v
git checkout core_v270rc2
# activate your conda (base) environment then install mamba if you don't have it with conda install -c conda-forge mamba
mamba env create -n coreRC2 -f environment.yml
conda activate core RC2
pip install -e .[develop]
you now should have a working esmvaltool with the Core pre-release candidate for testing; once your recipe has run, please tick the box in the comment above, if it failed, then please post a comment here with the full Traceback (either from on-screen or from the log) or attach the log. Very many thanks in advance :beer:
@ESMValGroup/esmvaltool-developmentteam this is me right now, after running more recipes than in my entire 5+ years of ESMValX development history put together :grin:
@remi-kazeroni @sloosvel running recipe recipe_climate_change_hotspot.yml
on Levante am getting this
2022-10-11 11:00:19,448 UTC [4189991] ERROR Missing data for preprocessor tas_cmip6_85/tas:
- Missing data for INM-CM4-8: tas
- Missing data for INM-CM5-0: tas
2022-10-11 11:00:19,448 UTC [4189991] ERROR Missing data for preprocessor pr_cmip6_85/pr:
- Missing data for INM-CM4-8: pr
- Missing data for INM-CM5-0: pr
- No input data available for years 2015-2100 in files:
/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ/CMIP/NCAR/CESM2/historical/r2i1p1f1/Amon/pr/gn/v20190401/pr_Amon_CESM2_historical_r2i1p1f1_gn_185001-201412.nc
am using Saskia's config user file - you guys know where the missing CMIP6 data may be by any chance? No way I can run it on JASMIN since it's missing OBS's - I have now turned on auto dll's let's see what's the hap with it, but do we tell DKRZ they're missing CMIP6 data or we just let it pass?
EDIT OK - it seems ESGF doesn't have them either:
2022-10-11 11:06:52,351 UTC [48716] ERROR No input files found for variable {'mip': 'Amon', 'short_name': 'pr', 'preprocessor': 'ensemble_members', 'variable_group': 'pr', 'diagnostic': 'pr_cmip6_85', 'project': 'CMIP6', 'exp': ['historical', 'ssp585'], 'dataset': 'INM-CM4-8', 'ensemble': 'r1i1p1f1', 'grid': 'gr', 'recipe_dataset_index': 25, 'timerange': '1980/2100', 'institute': ['INM'], 'activity': ['CMIP', 'ScenarioMIP'], 'alias': 'INM-CM4-8', 'original_short_name': 'pr', 'standard_name': 'precipitation_flux', 'long_name': 'Precipitation', 'units': 'kg m-2 s-1', 'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1980, 'end_year': 2100}
2022-10-11 11:06:52,352 UTC [48716] ERROR Looked for files matching
ESGF:/pr_Amon_INM-CM4-8_historical_r1i1p1f1_gr*.nc
ESGF:/pr_Amon_INM-CM4-8_ssp585_r1i1p1f1_gr*.nc
2022-10-11 11:06:52,352 UTC [48716] ERROR Set 'log_level' to 'debug' to get more information
2022-10-11 11:06:52,490 UTC [48716] ERROR No input files found for variable {'mip': 'Amon', 'short_name': 'pr', 'preprocessor': 'ensemble_members', 'variable_group': 'pr', 'diagnostic': 'pr_cmip6_85', 'project': 'CMIP6', 'exp': ['historical', 'ssp585'], 'dataset': 'INM-CM5-0', 'ensemble': 'r1i1p1f1', 'grid': 'gr', 'recipe_dataset_index': 26, 'timerange': '1980/2100', 'institute': ['INM'], 'activity': ['CMIP', 'ScenarioMIP'], 'alias': 'INM-CM5-0', 'original_short_name': 'pr', 'standard_name': 'precipitation_flux', 'long_name': 'Precipitation', 'units': 'kg m-2 s-1', 'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1980, 'end_year': 2100}
2022-10-11 11:06:52,490 UTC [48716] ERROR Looked for files matching
ESGF:/pr_Amon_INM-CM5-0_ssp585_r1i1p1f1_gr*.nc
ESGF:/pr_Amon_INM-CM5-0_historical_r1i1p1f1_gr*.nc
-> are we aware of this from the previous releases or this is something new?
@remi-kazeroni @sloosvel running recipe
recipe_climate_change_hotspot.yml
on Levante am getting this2022-10-11 11:00:19,448 UTC [4189991] ERROR Missing data for preprocessor tas_cmip6_85/tas: - Missing data for INM-CM4-8: tas - Missing data for INM-CM5-0: tas 2022-10-11 11:00:19,448 UTC [4189991] ERROR Missing data for preprocessor pr_cmip6_85/pr: - Missing data for INM-CM4-8: pr - Missing data for INM-CM5-0: pr - No input data available for years 2015-2100 in files: /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ/CMIP/NCAR/CESM2/historical/r2i1p1f1/Amon/pr/gn/v20190401/pr_Amon_CESM2_historical_r2i1p1f1_gn_185001-201412.nc
am using Saskia's config user file - you guys know where the missing CMIP6 data may be by any chance? No way I can run it on JASMIN since it's missing OBS's - I have now turned on auto dll's let's see what's the hap with it, but do we tell DKRZ they're missing CMIP6 data or we just let it pass?
EDIT OK - it seems ESGF doesn't have them either:
2022-10-11 11:06:52,351 UTC [48716] ERROR No input files found for variable {'mip': 'Amon', 'short_name': 'pr', 'preprocessor': 'ensemble_members', 'variable_group': 'pr', 'diagnostic': 'pr_cmip6_85', 'project': 'CMIP6', 'exp': ['historical', 'ssp585'], 'dataset': 'INM-CM4-8', 'ensemble': 'r1i1p1f1', 'grid': 'gr', 'recipe_dataset_index': 25, 'timerange': '1980/2100', 'institute': ['INM'], 'activity': ['CMIP', 'ScenarioMIP'], 'alias': 'INM-CM4-8', 'original_short_name': 'pr', 'standard_name': 'precipitation_flux', 'long_name': 'Precipitation', 'units': 'kg m-2 s-1', 'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1980, 'end_year': 2100} 2022-10-11 11:06:52,352 UTC [48716] ERROR Looked for files matching ESGF:/pr_Amon_INM-CM4-8_historical_r1i1p1f1_gr*.nc ESGF:/pr_Amon_INM-CM4-8_ssp585_r1i1p1f1_gr*.nc 2022-10-11 11:06:52,352 UTC [48716] ERROR Set 'log_level' to 'debug' to get more information 2022-10-11 11:06:52,490 UTC [48716] ERROR No input files found for variable {'mip': 'Amon', 'short_name': 'pr', 'preprocessor': 'ensemble_members', 'variable_group': 'pr', 'diagnostic': 'pr_cmip6_85', 'project': 'CMIP6', 'exp': ['historical', 'ssp585'], 'dataset': 'INM-CM5-0', 'ensemble': 'r1i1p1f1', 'grid': 'gr', 'recipe_dataset_index': 26, 'timerange': '1980/2100', 'institute': ['INM'], 'activity': ['CMIP', 'ScenarioMIP'], 'alias': 'INM-CM5-0', 'original_short_name': 'pr', 'standard_name': 'precipitation_flux', 'long_name': 'Precipitation', 'units': 'kg m-2 s-1', 'modeling_realm': ['atmos'], 'frequency': 'mon', 'start_year': 1980, 'end_year': 2100} 2022-10-11 11:06:52,490 UTC [48716] ERROR Looked for files matching ESGF:/pr_Amon_INM-CM5-0_ssp585_r1i1p1f1_gr*.nc ESGF:/pr_Amon_INM-CM5-0_historical_r1i1p1f1_gr*.nc
-> are we aware of this from the previous releases or this is something new?
You can run it with --skip-nonexistent, no problem. Also I don't think this is using any OBS, so you can run it on jasmin.
great, will do's, cheers @sloosvel :beer:
If the data is not on ESGF either, it may have been retracted?
In that case it should be removed from the recipe, as decided at the previous workshop and on it's way to the documentation but not quite merged yet in #2720.
@sloosvel you were right - that wretched recipe runs fine with --skip-nonexistent
, hefty bugger too
OK I am fairly happy with how our Core RC has been performing, if there are no major objections from the @ESMValGroup/technical-lead-development-team I shall release us a Core 2.7.0 stable tomorrow - lucky 13th :grin:
@ESMValGroup/esmvaltool-developmentteam Core's stable 2.7.0 has been released to PyPi earlier today, with the PR to conda-forge having just been merged now (the regro bot was ill all day until now) https://github.com/conda-forge/esmvalcore-feedstock/pull/50 - am closing this with many thanks to those who tested Core 2.7.0, and welcoming ESMValCore v2.7.0 to the world, a plucky little chap :grin:
@ESMValGroup/esmvaltool-developmentteam we now have a pre-release candidate for ESMValCore, namely RC2, which sits nicely on conda-forge and PyPi. For that purpose I have now created a special branch for testing ESMValTool with this pre-release candidate, namely https://github.com/ESMValGroup/ESMValTool/tree/core_v270rc2 which I am encouraging you to use to test your favourite recipe(s). As soon as the Github Actions finish (they are currently degraged yet again within 24h - GitHub's fault not ours) I will post the complete list of recipes that we will use to check boxes for recipes that run fine, or get to the bottom of any fail via GH issues. Stay tuned for the heads up soon! :beer:
Pro tip: make sure your intermediary cubes are not saved (check config) otherwise you will end up like me swearing at JASMIN's meager disk quota (which is OK, but not OK enough to save a bazillion intermediary cubes :laughing: )