search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
215 stars 126 forks source link

Preparing the release of v2.5.0 #2535

Closed remi-kazeroni closed 2 years ago

remi-kazeroni commented 2 years ago

@ESMValGroup/esmvaltool-developmentteam

The next release of ESMValTool and ESMValCore are approaching (see schedule). As part of the release process, a candidate release has been made for the Core (v2.5.0rc2) following the feature freeze announcement.

We have now tested all the ESMValTool recipes against the candidate release rc2 for the Core (see next post).

Please note that the feature freeze for the Tool is schedule for Monday, Feb 21. Please let us know as soon as possible if you are still working on something for this release by adding your issue or pull request to our v2.5.0 milestone: https://github.com/ESMValGroup/ESMValTool/milestone/8

remi-kazeroni commented 2 years ago

Recipe testing against v2.5.0rc2 of the Core

The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc2/ Among the 119 available recipes, 80 were ran successfully and 39 failed. A summary of the main failure reasons will be posted soon in this issue.

@ESMValGroup/esmvaltool-recipe-maintainers
Please check that your recipes ran successfully using the link above and let us know something needs to be corrected before the release. This includes that all plots are created and look fine, output files look fine, provenance is stored (checklist here and here).

Everyone can also help us by running a few recipes with the latest development installation in order to confirm these results.

Settings for the recipe testing Most recipes were run using the Cylc suite using max_parallel_tasks: 8, a time limit of 4h, and 16 GB of RAM. The more memory-intensive recipes were run on a full prepost node. All recipes were run on Mistral using offline=False to automatically download missing data. A total of 110 GB was downloaded (59 GB of CMIP6 data and 52 GB of CMIP5 data). We would need to understand why these data are not available on Mistral. This goes beyond the recent deletion of some CMIP5 data.

And special thanks to @bouweandela for making the recipe testing much more convenient 🥳

Edit: I reran 4 recipes using search_connection: url: "http://esgf-index1.ceda.ac.uk/esg-search" instead of url: http://esgf-data.dkrz.de/esg-search but that didn't help. The search through ESGF nodes takes more than the time limit (12 hours) or the data download still fails. These recipes are: mpqb/recipe_mpqb_xch4.yml, recipe_russell18jgr.yml, bock20jgr/recipe_bock20jgr_fig_8-10.yml, recipe_deangelis15nat.yml.

schlunma commented 2 years ago

Below we summarize all the issues found by the automatic run. Feel free to expand this table as you see fit.

Wontfix Issues

These issues will not be fixed in this release. They are generally due to data that has become unavailable or is not yet fully integrated into ESMValTool.

#### Missing data | Recipe | Reason of failure | Known problem? | Current Status | |--------|-------------------|----------------|----------------| | recipe_schlund20jgr_gpp_abs_rcp85 | missing CMIP5 data |retracted data|won't fix - ran successfully by @schlunma | | recipe_schlund20jgr_gpp_change_1pct | missing CMIP5 data |retracted data|won't fix - ran successfully by @schlunma| | recipe_schlund20jgr_gpp_change_rcp85 | missing CMIP5 data |retracted data|won't fix - ran successfully by @schlunma| | recipe_wenzel16nat | missing CMIP5 data |retracted data|won't fix| #### Other problems | Recipe | Reason of failure | Known problem? | Current Status | |--------|-------------------|----------------|----------------| | recipe_autoassess_radiation_rms_cfMon_all | `Coordinate plevs has var name plev7` |[#1238](https://github.com/ESMValGroup/ESMValCore/issues/1238)| will be fixed in v2.6|

Outstanding Issues

Missing data

ERA5
Recipe Missing variables Known problem? Current Status
recipe_check_obs rlns, rlus, rsns, rsus #1388
CMIP5
Recipe Reason of failure Known problem? Current Status
recipe_collins13ipcc missing CMIP5 data (areacello, rtmt, tas) #2408 (but more missing); Birgit: I commented out the models for which the "tas" and "rtmt" data were missing, I did not change anything related to the missing "areacello" data; an updated version of the recipe is available in the "fix_recipes_with_missing_data_branch"
recipe_flato13ipcc missing CMIP5 data (areacello) #2408
recipe_landcover missing CMIP5 data #2408; Birgit: I commented out the model for which the data is not available anymore; the updated recipe is available in the "fix_recipes_with_missing_data_branch"
recipe_recipe_perfmetrics_CMIP5 missing CMIP5 data (CESM1-CAM5-1-FV2: tas) #2408, Birgit: I commented out the one model for "tas" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch"
recipe_recipe_perfmetrics_CMIP5_4cds missing CMIP5 data (CESM1-CAM5-1-FV2: tas) #2408; Birgit: I commented out the one model for "tas" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch"
recipe_recipe_perfmetrics_CMIP5_land missing CMIP5 data (BNU-ESM: fgco2) #2408; Birgit: I commented out the one model for "fgco2" with missing data; an updated version of the recipe is available in the "fex_recipes_with_missing_data_branch"
recipe_seaice missing CMIP5 data (areacello) #2408 (but more missing)
recipe_wenzel14jgr missing CMIP5 data (fgco2, nbp) #2408 (but more missing); Birgit: I commented out the missing models in the recipe; there were already some models commented out, they are now marked with a double "##"; an updated version of the recipe is available in the "fix_recipes_with_missing_data_branch"
recipe_williams09climdyn missing CMIP5 data (sic) #2408; Birgit: 4 out of 5 datasets are not available; commenting all four out seems kind of wrong since then only one model (+ one CMIP6 model) would be left over to be used in the diagnostic; note by @axel-lauer: I think I vaguely remember that sic from CMIP5 AMIP runs was only available for very few models. As sic is prescribed in these runs, we used the same fields (from one model) for all others. So I guess this never worked in v2.0 and has to be remodeled. I would say let's postpone this for now.
CMIP6
Recipe Reason of failure Known problem? Current Status
recipe_climwip_brunner20esd missing CESM2 data (tas) Birgit: CESM2 data is available (on MISTRAL) but the specified ensembles for the ssp585 scenario are not; ensembles are specified as "ensemble: r(1:2)i1p1f1", but only r4i1p1f1, r10i1p1f1 and r11i1p1f1 are available
AUX
Recipe Reason of failure Known problem? Current Status
ESGF Data search / download incomplete
Recipe Reason of failure Known problem? Current Status
recipe_bock20jgr_fig_8-10 ESGF data request not finished after 12 hours Used to work before partial deletion of CMIP5 data at DKRZ
recipe_impact data download failure (EC-Earth3-Veg, areacella)
recipe_russell18jgr ESGF data request failure (twice the same)

Other problems

Diagnostic issue (Python)
Recipe Reason of failure Known problem? Current Status
Diagnostic issue (NCL)
Recipe Reason of failure Known problem? Current Status
Diagnostic issue (R)
Recipe Reason of failure Known problem? Current Status
recipe_extreme_events cdo remapcon
Core issue
Recipe Reason of failure Known problem? Current Status

Resolved Issues

#### Missing data | Recipe | Reason of failure | Known problem? | Current Status | |--------|-------------------|----------------|----------------| | recipe_anav13jclim | missing inmcm4 data (cVeg, cSoil) |#2408 EDIT by @schlunma: It looks like the data is available on [ESGF](https://esgf-data.dkrz.de/projects/esgf-dkrz/)|manual download of files| | recipe_check_obs | ESACCI-SEA-SURFACE-SALINITY/sos || fixed with ESMValGroup/ESMValTool#2542 | | recipe_deangelis15nat | ESGF data request not finished after 12 hours ||EDIT by @remi-kazeroni : recipe works for me if using `offline=true`; The fx files that are searched through ESGF nodes are actually not needed for the recipe run.| | recipe_mpqb_xch4 | data download failure (CESM2-WACCM, areacello) || EDIT by @axel-lauer: recipe works for me if I turn off automatic download in the user-config.yml; with automatic download (offline: false), recipe fails because of missing OFx files (areacello). | | recipe_ecs | missing ACCESS-ESM1-5 data (rsdt) | It looks like the data is available on [ESGF](https://esgf-data.dkrz.de/projects/esgf-dkrz/)|manual download of files; recipe ran fine| | recipe_hydro_forcing | MSWEP not available on Mistral |#2262|data manually downloaded from old source| | recipe_hype | missing shape file ||Missing files added to the central AUX folder on Mistral| | recipe_meehl20sciadv | missing ACCESS-ESM1-5 data (rsdt) | It looks like the data is available on [ESGF](https://esgf-data.dkrz.de/projects/esgf-dkrz/)|manual download of files; recipe ran fine| | recipe_schlund20esd | missing ACCESS-ESM1-5 data (rsdt) | It looks like the data is available on [ESGF](https://esgf-data.dkrz.de/projects/esgf-dkrz/)|manual download of files; recipe ran fine| | recipe_wflow | missing shape file |#2260|runs fine with the shapefile linked in #2260 | #### Other problems | Recipe | Reason of failure | Known problem? | Current Status | |--------|-------------------|----------------|----------------| | recipe_anav13jclim | `fatal: in time_operations (diag_scripts/shared/statistics.ncl), the first dimension of input is not time` (HWSD) |ESMValGroup/ESMValCore#1496|fixed by ESMValGroup/ESMValCore#1497| | recipe_autoassess_landsurface_soilmoisture | Climatology files missing | ESMValGroup/ESMValTool#2309 | tested successfully by V| | recipe_julia |||tested successfully by V| | recipe_kcs | issue with `years` |#2223| fixed with ESMValGroup/ESMValTool#2541 and ESMValGroup/ESMValCore#1502| | recipe_ocean_bgc | `volume statistics` |#2486|fixed by ESMValGroup/ESMValCore#1497| | recipe_ocean_bgc | `KeyError`: 'realms' in diagnostic |#2336|Fixed in ESMValGroup/ESMValTool#2540| | recipe_rainfarm |||tested successfully by V| | recipe_smpi_4cds | `multi_model_statistics` |ESMValGroup/ESMValCore#1499|solved by ESMValGroup/ESMValCore#1500| | recipe_arctic_ocean | issue with colormap ||Successfully ran by @schlunma| | recipe_carvalhais14nat | Matplotlib (norm, vmin, vmax) ||Successfully ran by @schlunma| | recipe_wenzel16jclim | NCL diagnostic hanging? || fixed by ESMValGroup/ESMValTool#2545 | | recipe_globwat | diagnostic hanging? (memory issue?) |#2543|fixed in ESMValGroup/ESMValTool#2548| | recipe_wenzel16nat | error in `carbon_ec/carbon_co2_cycle.ncl` - run hangs indefinitely |@axel-lauer: problem (1) missing data from MIROC-ESM; problem (2) preprocessed observational data (ESRL) contain only missing values| fixed by ESMValGroup/ESMValTool#2547 |
remi-kazeroni commented 2 years ago

The overview of issues found by running all recipes automatically is now complete (see post above). Please add already opened issues to the tables, we may have missed some.

It seems that the missing CMIP5 issue has slightly worsened since the last release (see ESMValGroup/ESMValTool#2408). But we were told that some deleted data will be retrieved from the archive soon.

We also have a few failures with the automatic download features (too long request on ESGF nodes, download failures, existing data but not found). Could someone else try to rerun some of the impacted recipes? I tried myself but got the same failures. I may not be using optimal settings for the automatic download. Any suggestion welcomed!

valeriupredoi commented 2 years ago

stellar work, guys! What's the actual problem with the two Julia recipes? For recipe_autoassess_landsurface_soilmoisture we have https://github.com/ESMValGroup/ESMValTool/issues/2309 - but should work on JASMIN where those files currently live, I'll have a test now :+1:

valeriupredoi commented 2 years ago

also, I'm gonna run a bunch of those CMIP5 missing recipes since we have CMIP5 on JASMIN (unlike some German HPC that did a terribly ungerman thing and deleted all that data :rofl: )

valeriupredoi commented 2 years ago

OK here's me stuffs on JASMIN:

valeriupredoi commented 2 years ago

let me know if you guys want me to run any others but bugger me flat, am missing lots of OBS's at JASMIN :mag:

remi-kazeroni commented 2 years ago

What's the actual problem with the two Julia recipes?

Here is the type of error I am getting. Did I do something wrong when installing Julia? I simply followed the instructions and did esmvaltool install Julia which was successful.

valeriupredoi commented 2 years ago

you need to purge the (most probably existing) ~/.julia dir, re-create the env so Julia can be installed cleanly from conda-forge, install esmvaltool, then esmvaltool install Julia

remi-kazeroni commented 2 years ago

We had two cases where using offline=true instead of offline=false was enough to run the recipe successfully: recipe_deangelis15nat.yml and recipe_mpqb_xch4.yml. In both cases, all the files needed are available in the official pool of CMIP5-6 data on Mistral and no data download is needed. Nevertheless, if the automatic download option is used, the tool will search for the many missing fx files on ESGF nodes in order to download these files which are later not used in the recipe. In particular the search for fx files took more than 12 hours (wall time on Mistral) for recipe_deangelis15nat.yml while the run with offline=true took 1 hour. Would this deserves a separate issue for further discussion?

valeriupredoi commented 2 years ago

yes - very good find, Remi! That's a pretty serious performance bug :bug:

schlunma commented 2 years ago

I agree that this is definitely something we need to check, but we probably can't fix this in time for v2.5. I suggest adding this to v2.6.

valeriupredoi commented 2 years ago

am opening an issue as we speak

valeriupredoi commented 2 years ago

also, I ran recipe_seaice.yml on JASMIN:

2022-02-16 16:48:03,524 UTC [20412] ERROR   Missing data for preprocessor seaice_tsline/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR   Missing data for preprocessor seaice_trends/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR   Missing data for preprocessor seaice_ecs/areacello:
- Missing data for CMIP5_bcc-csm1-1-m_historical_r0i0p0: areacello
- Missing data for CMIP5_bcc-csm1-1-m_rcp85_r0i0p0: areacello
- Missing data for CMIP5_inmcm4_historical_r0i0p0: areacello
2022-02-16 16:48:03,524 UTC [20412] ERROR   Missing data for preprocessor seaice_yod/areacello:
- Missing data for bcc-csm1-1-m_historical: areacello
- Missing data for inmcm4_historical: areacello

this is like shooting ducks with ballistic missiles: MRI has areacello in historical only but BCC has the same fx variable in piControl only - if we don't get the wildcard search in we'll always hit these sort of roadblocks; esgf-pyclient has no chance to retrieve it either if the same crappe is at other ESGF nodes too, it will just end up spending time looking around

bettina-gier commented 2 years ago

@ESMValGroup/esmvaltool-recipe-maintainers Please check that your recipes ran successfully using the link above and let us know something needs to be corrected before the release. This includes that all plots are created and look fine, output files look fine, provenance is stored (checklist here and here).

Should we do a comment with the list of full recipes and a checkbox for having them checked for the scientific ok - provenance and figures looking fine? Otherwise we never know if anyone actually looked at the output for the recipes that ran fine.

valeriupredoi commented 2 years ago

@bettina-gier yes that is a good idea always - maybe this can be done during the freeze, like we did during other releases :beer:

schlunma commented 2 years ago

I just tried to identify the issue with recipe_smpi_4cds.yml. I think this might be related to memory issues: when using only the years 2001-2005 the diagnostic ua (all other diagnostics deactivated) the recipe runs fine, when using 2000-2005 (or the original range 1986-2005) the following error appears:

Traceback (most recent call last):
...
  File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 468, in multi_model_statistics
    group_statistics = _multiproduct_statistics(
  File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 370, in _multiproduct_statistics
    statistics_cubes = _multicube_statistics(cubes=cubes,
  File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 351, in _multicube_statistics
    result_cube = _compute_eager(aligned_cubes,
  File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 286, in _compute_eager
    single_model_slices = [cube[chunk] for cube in cubes]
  File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 286, in <listcomp>
    single_model_slices = [cube[chunk] for cube in cubes]
  File "/work/bd0854/b309192/soft/miniconda3/envs/core_rc2/lib/python3.10/site-packages/iris/cube.py", line 2323, in __getitem__
    dimension_mapping, data = iris.util._slice_data_with_keys(
  File "/work/bd0854/b309192/soft/miniconda3/envs/core_rc2/lib/python3.10/site-packages/iris/util.py", line 779, in _slice_data_with_keys
    raise IndexError("Cannot index with zero length slice.")

see https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc2/recipe_smpi_4cds_20220213_125027/run/main_log_debug.txt.

The weird thing is that I'm running this on a 256 GB node, and the output says Maximum memory used (estimate): 48.4 GB. Note that this is 4D (time, plev, lat, lon) data, so memory issues would not be surprising.

valeriupredoi commented 2 years ago

no, it says Maximum memory used (estimate): 187.4 GB - but don't matter, if it was memory it would have said MemoryError :grin: It looks like _compute_slices() returns 0-len slices, it could be that n_slices = int(np.ceil(total_bytes / gigabyte)) is pish - something with too little memory estimated?

schlunma commented 2 years ago

Yes, you're right!! It is related to this. Here is the output of some prints:

n_timesteps: 71
slices_len: 8
n_slices: 10

start, end: 0 8
start, end: 8 16
start, end: 16 24
start, end: 24 32
start, end: 32 40
start, end: 40 48
start, end: 48 56
start, end: 56 64
start, end: 64 71
start, end: 72 71

The last indices are definitely not correct. I will try to fix this.

(the 48.4 GB referred to my run with only 6 year; the 187.4 GB correspond to using all 20 years)

remi-kazeroni commented 2 years ago

Status of the release process and recipe testing:

Using the release candidate v2.5rc2 for the Core, we initially got 80 of 119 public recipes to run successfully during the automated testing. Over the past week, the number of successful recipes was increased to 96 by:

The latest fixes to the Core warrants another (an probably last) release candidate for the Core. @schlunma will soon release v2.5rc3. This will be done at the same time as the feature freeze for the Tool. We will perform another round of recipe testing and ask maintainers to check the output later this week.

remi-kazeroni commented 2 years ago

Recipe testing against v2.5.0rc3 of the Core

The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc3/ Among the 120 available recipes, 96 were ran successfully and 24 failed.

Given the current uncertainties in the release process and the choice of iris version used for v2.5 (see https://github.com/ESMValGroup/ESMValCore/issues/1509), we may not need to ask recipe maintainers to check the results of their recipes at this stage. We will do so for the next round of recipe testing (most likely needed if iris=3.1 is chosen) or contact maintainers to check this round of recipe testing later on. Nevertheless, feel all free to take a look at your favourite recipes.

Summary of the main issues

6 recipes ran successfully with v2.5.0rc2 but not this time:

Among the 18 other failing recipes, 1 "new" issue discovered:

The 17 other failures remain unchanged:

remi-kazeroni commented 2 years ago

I wonder if we could speed-up the search on ESGF nodes for some recipes (recipe_anav13jclim, recipe_climwip_brunner2019_med, recipe_schlund20esd, recipe_bock20jgr_fig_8-10, but not only) by following this suggestion from @bouweandela regarding areacella. Of course, this wouldn't not apply to all cases, but it could help to save computing resources.

schlunma commented 2 years ago
valeriupredoi commented 2 years ago

cheers @remi-kazeroni and @schlunma - for the Julia ones, have you nuked the presumably existent ~/.julia dir and still they don't run properly (after a reinstall with esmvaltool install Julia)? Yes, the Autoassess recipes run fine on JASMIN :+1:

remi-kazeroni commented 2 years ago

Recipe testing against v2.5.0rc4 of the Core

The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc4/debug.html You can also have a look at the recipe portal to browse these results: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc4/

Among the 121 available recipes, 111 were run successfully and only 10 failed. This is the result of fixes to the Core, recipes/diagnostics and chasing missing data. A summary of the remaining failures in the next post.

This last round of recipe testing also shows nice improvement in the automatic download of data thanks to the updated esgf-pyclient package. The data search through ESGF nodes takes only minutes in cases where it used to take hours to find all the fx files (see https://github.com/ESMValGroup/ESMValCore/issues/1495). Thanks for your great work @bouweandela!

@ESMValGroup/esmvaltool-recipe-maintainers, it would be great if you could check the output of the recipes you are responsible for and tick the corresponding box if happy with the output (checklist here and here). If something does not look right, please let us know if this issue.

List of recipes:

remi-kazeroni commented 2 years ago

Summary of the main issues

valeriupredoi commented 2 years ago

excellent work guys! I'll test the AA recipes on JASMIN, so we have only 8 crappe'd :grin:

valeriupredoi commented 2 years ago

the cfMon_all AA recipe dies with the usual

esmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp:
 Coordinate plevs has var name plev7 instead of plev
 tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not None
 clisccp: does not match coordinate rank
in cube:

issue coz of the ISCCP data on JASMIN - I can turn that dataset off, but I know for a fact that that should work if we fix the ISCCP data, what do you recommend @remi-kazeroni ? The other AA went through no problemo

remi-kazeroni commented 2 years ago

the cfMon_all AA recipe dies with the usual

esmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp:
 Coordinate plevs has var name plev7 instead of plev
 tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not None
 clisccp: does not match coordinate rank
in cube:

issue coz of the ISCCP data on JASMIN - I can turn that dataset off, but I know for a fact that that should work if we fix the ISCCP data, what do you recommend @remi-kazeroni ? The other AA went through no problemo

Thanks for double checking the AA recipes @valeriupredoi! We already have an issue open for the problematic dataset: https://github.com/ESMValGroup/ESMValCore/issues/1238 If it's better to keep the dataset in the recipe, I'd suggest to address that for the next release 👍

valeriupredoi commented 2 years ago

awesome, yeah - I believe I opened that issue judging by the title containing semi-expletives :sweat_smile:

remi-kazeroni commented 2 years ago

Status of the recipe testing using the version v2.5.0rc4 for the Core: there are up to 115 recipes that can be run successfully. Among the remaining 6 recipes, there should not be any missing data problem any more (CMIP5, CMIP6, OBS,...). For each failing recipe, there is a dedicated issue or PR to fix it. In the (hopefully 😆) last round of testing, we need to pay closer attention to the recipe_perfmetrics_land_CMIP5 (model fix needed?) and the recipe_collins13ipcc. For the later one, there is probably not enough memory on the Mistral nodes to run it. But maybe it could work with Levante, worth trying.

valeriupredoi commented 2 years ago

@remi-kazeroni stellar work! Do we have a mention in the documentation of that recipe that needs gargantuan amounts of memory?

remi-kazeroni commented 2 years ago

For information, our ESMValTool project data from Mistral will be copied to Levante (successor machine) on March 14. This comprises user data (e.g. /work/bd0854/b3*****) but also OBS/RAWOBS data and data from different projects. After the copy, DKRZ will try to keep the data synchronize between the 2 machines until March 22. After that, it will be our responsibility to copy data from Mistral to Levante. More info here: https://docs.dkrz.de/blog/2022/mistral-levante-data-move.html

I think this will require some actions from us to adjust the data paths, reorganize the directory structure of the group space... But for the time being, I'm hoping that we could finish the recipe testing and the release on Mistral only and configure things on Levante once the release is over. So hopefully we can finish the release sometime next week at the latest :)

remi-kazeroni commented 2 years ago

Recipe testing against v2.5.0rc6 of the Core

The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc6/debug.html You can also have a look at the recipe portal to browse these results: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc6/

Among the 121 available recipes, 114 were run successfully and only 7 failed. Since the outcome of this round of testing is identical to the previous one, there is no need for another thorough recipe output checking. Nevertheless, feel free to check the output of your favourite recipe if you wish. I think this round of testing is good enough to do the release next week 🍻 No new bug was discovered during this testing. Among the 7 known issues, we have:

valeriupredoi commented 2 years ago

make that 6 now :grin: so that's a 5% loss ratio - am very happy and it's a good start for my day off today - excellent work the Release Boys :beer:

remi-kazeroni commented 2 years ago

I forgot to write that I needed to remove some output files (mostly netcdf files) for the recipe runs that required lots of disk space. The reason is that the recipe outputs are copied to the disk of the bot VM. Disk space there is limited to 1TB while some recipes produce 10GB - 100GB of data (even with the /preproc direcrotry removed). In some cases, there was not much left to check as some recipes do not produce plots but large nc files.

To help checking recipes, I have rerun 2 recipes successfully: recipe_wflow (all netcdf files kept) and recipe_daily_era5 (some netcdf files kept). @axel-lauer: do you want to have another look at these 2 recipes?

axel-lauer commented 2 years ago

@axel-lauer: do you want to have another look at these 2 recipes?

Thanks @remi-kazeroni. As far as I can tell, the output of the two recipes looks OK. I updated the list above.

schlunma commented 2 years ago

Closing this now since v2.5.0 is out :tada: Thanks to everyone who contributed to this!

An issue is open for every failing recipe; let's move the discussion to these. Cheers!!