search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
223 stars 128 forks source link

Test before ESMValTool v2.1.0 release #1875

Closed valeriupredoi closed 4 years ago

valeriupredoi commented 4 years ago

Hello @ESMValGroup/esmvaltool-developmentteam - it's me the release manager asking you a big favour - can peeps grab their favorite recipe(s) and run it with the release branch and if so tick the box(es) below, please? We will be releasing ESMValTool v2.1.0 on next Monday (26 October) and it would be great to have tested all these recipes to be sure all works well. @mattiarighi and @hb326 it would also be fantastic if you guys had a bit of time to test some of the cmorizers, since I have no access to any of the raw data. Your efforts shall be forever appreciated and we promise we'll fix a robot to do this disrt work before the next release :beer: :beer:

cmorizers:

examples:

hydrology:

schlund20jgr:

valeriupredoi commented 4 years ago

hey guys @ESMValGroup/esmvaltool-developmentteam we'd really appreciate it if you got your favorite recipe run and ticked the box - just a couple days til the release and if these things don't get tested before the day of the release we'll exclude your recipe that wasn't tested (just kidding...or maybe not :grin: ) Cheers muchly! :beer:

Peter9192 commented 4 years ago

@jeromaerts would you have time to run some simple hydrology recipes with this branch?

remi-kazeroni commented 4 years ago

I successfully ran all the recipes in examples except the following ones:

Could someone doublecheck for these?

ruthlorenz commented 4 years ago

recipe_extract_shape.yml does not work for me either. recipe header says "The shapefile can be copied from esmvaltool/diag_scripts/shapeselect/testdata/Elbe.shp and placed in the auxiliary_data_dir defined in config-user.yml."

I did that, but then it crashes with "fiona.errors.DriverError: Unable to open /ESMVal/auxiliary_data/Elbe.shx or /ESMVal/auxiliary_data/Elbe.SHX. Set SHAPE_RESTORE_SHX config option to YES to restore or create it."

recipe_julia.yml works fine for me (ticked it above).

The other two I cannot check because of missing data.

ruthlorenz commented 4 years ago

I also run most of recipe_collins13ipcc.yml I cannot check one diagnostic in it because of missing obs data (the second last, fig12-31ad). Rest runs fine just takes forever (my bad so I should not complain ).

bouweandela commented 4 years ago

@ruthlorenz You need to copy all the Elbe.* files, not just the one with the .shp extension.

valeriupredoi commented 4 years ago

@ruthlorenz @remi-kazeroni cheers for the help guys! Ruth, as Bouwe says - it needs all the metadata files for the Elbe region, shapefiles come as a collection of files for each of the region they describe. @bouweandela you got time to check the other non-obvious fails, Jasmin is belly up today, great timing for it to go

bouweandela commented 4 years ago

recipe_check_obs.yml (preprocessor issue) recipe_preprocessor_derive_test.yml ((same?) preprocessor issue)

@remi-kazeroni Could you attach the file run/main_log_debug.txt so we can see what's going wrong?

remi-kazeroni commented 4 years ago

recipe_check_obs.yml (preprocessor issue) recipe_preprocessor_derive_test.yml ((same?) preprocessor issue)

@remi-kazeroni Could you attach the file run/main_log_debug.txt so we can see what's going wrong?

main_log_debug_recipe_check_obs.txt main_log_debug_recipe_preprocessor_derive_test.txt

and the recipes I used (one or two missing datasets commented out)

recipe_check_obs.yml.txt recipe_preprocessor_derive_test.yml.txt

ruthlorenz commented 4 years ago

My bad, copying all files helps but recipe still fails.

main_log_debug.txt log.txt

SarahAlidoost commented 4 years ago

I am working on "cmorizers: recipe_era5.yml", after that I can test some of the hydrology recipes. @Peter9192 and @jeromaerts FYI.

valeriupredoi commented 4 years ago

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

valeriupredoi commented 4 years ago

@ruthlorenz cheers for your efforts - I have reproduced your issue with recipe_extract_shape.yml and will look into it :beer:

valeriupredoi commented 4 years ago

OK @bouweandela - as recipe admin - you have a bug, sir :grin: https://github.com/ESMValGroup/ESMValTool/issues/1878

remi-kazeroni commented 4 years ago

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

@valeriupredoi you are right, I didn't have the lastest master branch, sorry... I have retested all the example recipes and I still could not run the same two ones, namely: recipe_check_obs.yml and recipe_preprocessor_derive_test.yml. These stop at the same point as before, including the PIOMAS task for recipe_check_obs.yml. Could it be due to mssing data on DKRZ?

valeriupredoi commented 4 years ago

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

@valeriupredoi you are right, I didn't have the lastest master branch, sorry... I have retested all the example recipes and I still could not run the same two ones, namely: recipe_check_obs.yml and recipe_preprocessor_derive_test.yml. These stop at the same point as before, including the PIOMAS task for recipe_check_obs.yml. Could it be due to mssing data on DKRZ?

I too don't have the full data on Jasmin, but mines stop at Missing data, as it should be the case when there's no data. I did manage to run the PIOMAS though since there is data on Jasmin (albeit not correctly named in file names); I am bit confused how @mattiarighi ran them last time (last release) - Mattia, any chance you can try them yourself again, mate? Sorry, I know you busy :beer:

katjaweigel commented 4 years ago

I found an issue with recipedeangelis15nat.yml: The recipe is running, but several figures are not displayed correctly (deangelisf3f4/deangelisf3f4/fig3a.png, deangelisf3f4/deangelisf3f4/fig3b.png, and deangelisf3f4/deangelisf3f4/fig4.png). Looks like an issue with the x axis limits. The recipe is unchanged since the last test on 7 October. I'll try to find the reason tomorrow.

valeriupredoi commented 4 years ago

cheers @katjaweigel - it could be a different version of whatever package is used to plot the plots? :beer:

bouweandela commented 4 years ago

@remi-kazeroni The recipes stop because there is not a single dataset defined for the variable you're trying to process, because you commented the one dataset that was there out. The error message is not very informative unfortunately, I've created an issue to report it https://github.com/ESMValGroup/ESMValCore/issues/843.

koldunovn commented 4 years ago

recipe_arctic_ocean.yml runs, but with one caveat :) After environment update I get matplotlib 3.3.1, that has a bug, that affects the plotting (https://github.com/mwaskom/seaborn/issues/2194). This was fixed in matplotlib 3.3.2, so with this version everything runs smoothly. You might consider to put matplotlib!=3.3.1 in setup.py.

dr-ko commented 4 years ago

recipe_carvalhais14nat.yml runs.

katjaweigel commented 4 years ago

I had a closer look at the figures and it is not the axis but the value which changed (factor 100.0) for the plots from recipe_deangelis15nat.yml. Has any unit changed from 1 to % ? (It's in a variable calculated from several other ones, including prw and radiance units but maybe also others. I'll check which of them could have changed.)

bouweandela commented 4 years ago

Has any unit changed from 1 to % ?

@katjaweigel I suspect this might be related to https://github.com/ESMValGroup/ESMValCore/pull/754

axel-lauer commented 4 years ago

I just successfully tested the full examples/recipe_check_obs.yml on Mistral (DKRZ).

katjaweigel commented 4 years ago

@bouweandela thanks, yes it looks like there is a change in the derived variable rsnstcsnorm (which is in %) it is already too high by the factor 100 in the preprocessed data. In _derive/rsnstcsnorm.py I calculated manually times 100.0, I guess this is now done automatically in addition? I'll remove the manual factor and try if it changes the plots.

bouweandela commented 4 years ago

I guess this is now done automatically in addition?

@schlunma Could you answer that question?

valeriupredoi commented 4 years ago

@schlunma is on hols, he ain't gonna reply until the week after, looking through the PR it looks like the % units are now updated automatically :+1:

katjaweigel commented 4 years ago

@schlunma and @bouweandela Adding rsnstcsnorm_cube.units = Unit('percent') in ESMValCore/esmvalcore/preprocessor/_derive/rsnstcsnorm.py solves the issue with the factor 100.0 (or leaving the factor 100.0 out, but I think the other one is more "backward compatible"). But this is in the Core - I guess it should nevertheless be fixed now instead of putting some work around into the diagnostic?

valeriupredoi commented 4 years ago

cheers @koldunovn for both running the test and for pointing to us the Matplotlib bug - I don't think there's a need to restrict the version since 3.3.2 has been out and about for more than a month now (see release table ) and any newer environment will pick up 3.3.2 no problemo :+1:

valeriupredoi commented 4 years ago

@katjaweigel the units should be set correctly in Core from PR ESMValGroup/ESMValCore#754 so technically no need to set them - is this a case that's missed by the said PR?

katjaweigel commented 4 years ago

@valeriupredoi They are set correctly, but to calculate percentage I always had the factor 100.0 in there (but I forgot to set the unit manually to percentage). So the old version was: rsnstcsnorm_cube = (((rsdt_cube - rsutcs_cube) - (rsdscs_cube - rsuscs_cube)) / rsdt_cube) * 100.0 (all cubes despite the result with the same radiance unit). As long as it did not care about the unit this worked well for my further calculations. Now it tries to be clever and somehow decided that dividing cubes with the same unit through each other should result in percentage. This puts the unit right but destroyed the value, because it is not clever enough to see that the factor 100.0 was already there. So there are two possible solution: 1) remove the factor 100.0 and let it correct everything itself 2) set the unit and the factor 100.0 manually, so the new cube is correct from the beginning and does not need the automatic fix anymore. (I tested it, both works). Because it could be possible that somebody has a strange combination of new Core and old Tool version I'd prefer the solution (2), because it works both with and without the new automatic fix.

koldunovn commented 4 years ago

cheers @koldunovn for both running the test and for pointing to us the Matplotlib bug - I don't think there's a need to restrict the version since 3.3.2 has been out and about for more than a month now (see release table ) and any newer environment will pick up 3.3.2 no problemo 👍

Well, I did an update with conda env update --name esmvaltool --file environment.yml and I got the 3.3.1, so maybe some dependency is more conservative on the matplotlib site. But of course in a few months the chances to hit this problem will be fewer :)

valeriupredoi commented 4 years ago

@koldunovn - good point! conda update works in mysterious ways, conda env create picked up 3.3.2 for me so that's why I'm not too concerned @katjaweigel cheers! I would not change the current Core - but is the resulting cube unit from the derivation script wrong? If it's wrong then it needs to be fixed in there and not by the external fix :beer:

katjaweigel commented 4 years ago

The resulting value is now wrong (with the factor 100.0 because it is used twice). Therefore I'd like to add rsnstcsnorm_cube.units = Unit('percent') in ESMValCore/esmvalcore/preprocessor/_derive/rsnstcsnorm.py then the values for rsnstcsnorm_cube are correct again

valeriupredoi commented 4 years ago

good stuff @katjaweigel - could you pls open a PR for that in le Core :beer:

katjaweigel commented 4 years ago

ok

valeriupredoi commented 4 years ago

ok

Danke schoen! :beer:

katjaweigel commented 4 years ago

@valeriupredoi Should I use master or release_2.1 as base?

valeriupredoi commented 4 years ago

for Core, use master please, we've made the release and we should delete that branch :beer:

SarahAlidoost commented 4 years ago

I am working on "cmorizers: recipe_era5.yml", after that I can test some of the hydrology recipes. @Peter9192 and @jeromaerts FYI.

For these recipes from hydrology folder run was successful:

I didn't run recipe_hype.yml, because I am looking for the shapefile needed by the recipe.

For cmorizer: recipe_era5.yml, some files are still downloading. Also, I'm working on setting the correct default drs, see this issue.

soufianekar commented 4 years ago

successfully tested recipe_cvdp.yml

Peter9192 commented 4 years ago

I wonder why the test of examples/recipe_python crashed in this bot test: https://github.com/ESMValGroup/ESMValTool/pull/1882

bouweandela commented 4 years ago

Successfully tested examples/recipe_julia.yml

earnone commented 4 years ago

We have an issue with the plotting part of recipe_extreme_events.yml. @jhardenberg is looking into that.

valeriupredoi commented 4 years ago

@Peter9192 - am running that myself; @earnone will run it too. BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions :+1:

valeriupredoi commented 4 years ago

OK examples/recipe_python.yml tested successfully :+1:

Peter9192 commented 4 years ago

BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions

There is an issue about this in the bot repo.

I managed to reproduce the examples/recipe_python.yml errors on a fresh install from the release branch. Then, upgrading cartopy to 0.18.0 fixed the issue for me.

bouweandela commented 4 years ago

BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions

This information would be so useful for debugging, that we could also consider including it in the debug log: https://github.com/ESMValGroup/ESMValCore/issues/847

Peter9192 commented 4 years ago

https://github.com/SciTools/cartopy/issues/1615 btw this was the cartopy issue that we had in the example recipe

valeriupredoi commented 4 years ago

OK so this seems to be a bit of an endemic (not pandemic :grin: ) issue - reported above by @koldunovn (Matplotlib in that case) - either a conda update or just a simple fresh install will not cut it, recreating the env from scratch will do (as in my case, I built my env a few days ago, so fairly new) - it looks like a few dependencies have now become very sensitive to versioning and we must rebuild the env