Final test for v2.0.0 release

mattiarighi commented 4 years ago

A checklist of the ESMValTool recipes to be tested before the release

[x] recipe_era5.yml
[x] recipe_check_obs.yml
[x] recipe_concatenate_exps.yml
[x] recipe_correlation.yml
[x] recipe_extract_shape.yml
[x] recipe_julia.yml
[x] recipe_my_personal_diagnostic.yml
[x] recipe_ncl.yml
[x] recipe_preprocessor_derive_test.yml
[x] recipe_preprocessor_test.yml
[x] recipe_python_object_oriented.yml
[x] recipe_python.yml
[x] recipe_r.yml
[x] recipe_variable_groups.yml
[x] recipe_hype.yml
[x] recipe_lisflood.yml
[x] recipe_marrmot.yml
[x] recipe_pcrglobwb.yml
[x] recipe_wflow.yml
[x] recipe_albedolandcover.yml
[x] recipe_anav13jclim.yml
[x] recipe_arctic_ocean.yml
[x] recipe_autoassess_landsurface_permafrost.yml
[x] recipe_autoassess_landsurface_snow.yml
[x] recipe_autoassess_landsurface_soilmoisture.yml
[x] recipe_autoassess_landsurface_surfrad.yml
[x] recipe_autoassess_radiation_rms_Amon_all.yml
[x] recipe_autoassess_radiation_rms_Amon_obs.yml
[x] recipe_autoassess_radiation_rms_cfMon_all.yml
[x] recipe_autoassess_stratosphere.yml
[x] recipe_capacity_factor.yml
[x] recipe_carvalhais14nat.yml
[x] recipe_clouds_bias.yml
[x] recipe_clouds_ipcc.yml
[x] recipe_collins13ipcc.yml
[x] recipe_combined_indices.yml
[x] recipe_consecdrydays.yml
[x] recipe_cox18nature.yml
[x] recipe_cvdp.yml
[x] recipe_deangelis15nat.yml
[x] recipe_diurnal_temperature_index.yml
[x] recipe_ecs_constraints.yml
[x] recipe_ecs_scatter.yml
[x] recipe_ecs.yml
[x] recipe_ensclus.yml
[x] recipe_extreme_events.yml
[x] recipe_extreme_index.yml
[x] recipe_eyring06jgr.yml
[x] recipe_flato13ipcc.yml
[x] recipe_heatwaves_coldwaves.yml
[x] recipe_hyint_extreme_events.yml
[x] recipe_hyint.yml
[x] recipe_landcover.yml
[x] recipe_lauer13jclim.yml
[x] recipe_li17natcc.yml
[x] recipe_martin2018grl.yml
[x] recipe_miles_block.yml
[x] recipe_miles_eof.yml
[x] recipe_miles_regimes.yml
[x] recipe_modes_of_variability.yml
[x] recipe_multimodel_products.yml
[x] recipe_ocean_amoc.yml
[x] recipe_ocean_bgc.yml
[x] recipe_ocean_example.yml
[x] recipe_ocean_ice_extent.yml
[x] recipe_ocean_Landschuetzer2016.yml
[x] recipe_ocean_multimap.yml
[x] recipe_ocean_quadmap.yml
[x] recipe_ocean_scalar_fields.yml
[x] recipe_perfmetrics_CMIP5_4cds.yml
[x] recipe_perfmetrics_CMIP5.yml
[x] recipe_perfmetrics_land_CMIP5.yml
[x] recipe_quantilebias.yml
[x] recipe_rainfarm.yml
[x] recipe_runoff_et.yml
[x] recipe_russell18jgr.yml
[x] recipe_seaice_drift.yml
[x] recipe_seaice_feedback.yml
[x] recipe_seaice.yml
[x] recipe_shapeselect.yml
[x] recipe_smpi_4cds.yml
[x] recipe_smpi.yml
[x] recipe_snowalbedo.yml
[x] recipe_spei.yml
[x] recipe_tcr.yml
[x] recipe_thermodyn_diagtool.yml
[x] recipe_toymodel.yml
[x] recipe_validation_CMIP6.yml
[x] recipe_validation.yml
[x] recipe_wenzel14jgr.yml
[x] recipe_wenzel16jclim.yml
[x] recipe_wenzel16nat.yml
[x] recipe_williams09climdyn_CREM.yml (could only be tested with 1 dataset because of missing data)
[x] recipe_zmnam.yml

And here the same for the cmorizers:

[x] cmorize_obs_aphro_ma.py
[x] cmorize_obs_aura_tes.ncl
[x] cmorize_obs_berkeleyearth.py
[x] cmorize_obs_calipso_goccp.ncl
[x] cmorize_obs_cds_satellite_albedo.py
[x] cmorize_obs_cds_satellite_lai_fapar.py
[x] cmorize_obs_cds_satellite_soil_moisture.ncl
[x] cmorize_obs_cds_uerra.py
[x] cmorize_obs_cds_xch4.ncl
[x] cmorize_obs_cds_xco2.ncl
[x] cmorize_obs_ceres_ebaf.py
[x] cmorize_obs_ceres_syn1deg.ncl
[x] cmorize_obs_cowtanway.py
[x] cmorize_obs_cru.py
[x] cmorize_obs_ct2019.py
[x] cmorize_obs_duveiller2018.py
[x] cmorize_obs_e_obs.py
[x] cmorize_obs_eppley_vgpm_modis.py
[x] cmorize_obs_era_interim_land.py
[x] cmorize_obs_era_interim.py
[x] cmorize_obs_esacci_aerosol.ncl
[x] cmorize_obs_esacci_cloud.ncl
[x] cmorize_obs_esacci_fire.ncl
[x] cmorize_obs_esacci_landcover.ncl
[x] cmorize_obs_esacci_oc.py
[x] cmorize_obs_esacci_ozone.ncl
[x] cmorize_obs_esacci_soilmoisture.ncl
[x] cmorize_obs_esacci_sst.ncl
[x] cmorize_obs_ersl.ncl
[x] cmorize_obs_fluxcom.py
[x] cmorize_obs_gcp.py
[x] cmorize_obs_ghcn_cams.py
[x] cmorize_obs_ghcn.ncl
[x] cmorize_obs_gistemp.py
[x] cmorize_obs_gpcc.py
[x] cmorize_obs_hadcrut3.ncl
[x] cmorize_obs_hadcrut4.ncl
[x] cmorize_obs_hadisst.ncl
[x] cmorize_obs_haloe.ncl
[x] cmorize_obs_hwsd.py
[x] cmorize_obs_isccp_fh.ncl
[x] cmorize_obs_jma_transcom.py
[x] cmorize_obs_lai3g.py
[x] cmorize_obs_landflux_eval.py
[x] cmorize_obs_landschuetzer2016.py
[x] cmorize_obs_mac_lwp.ncl
[x] cmorize_obs_merra2.py
[x] cmorize_obs_mls_aura.py
[x] cmorize_obs_modis.ncl
[x] cmorize_obs_mte.py
[x] cmorize_obs_ncep.ncl
[x] cmorize_obs_ndp.py
[x] cmorize_obs_niwa_bs.ncl
[x] cmorize_obs_nsidc_0116_nh.py
[x] cmorize_obs_nsidc_0116_sh.py
[x] cmorize_obs_osi_450_nh.py
[x] cmorize_obs_osi_450_sh.py
[x] cmorize_obs_patmos_x.ncl
[x] cmorize_obs_persiann_cdr.py
[x] cmorize_obs_phc.py
[x] cmorize_obs_piomas.py
[x] cmorize_obs_regen.py
[x] cmorize_obs_uwisc.ncl
[x] cmorize_obs_woa.py

Should we also test the download scripts?

mattiarighi commented 4 years ago

@jvegasbsc how do you run the cmorizers with the new interface?

valeriupredoi commented 4 years ago

like this @mattiarighi (not sure about tcsh, this is for bash only though):

if [ $"day" = "Saturday" ]; then
    cmorizers="Mattia shouldn't work"
else
    cmorizers="Broken, see below"
fi

and the broken bit is because:

(esmvaltool_users) [valeriu@jasmin-sci1 ~]$ cmorize_obs -o WOA --config-file esmvaltool_var_test/config-user.yml 
Traceback (most recent call last):
  File "/home/users/valeriu/anaconda3R/envs/esmvaltool_users/bin/cmorize_obs", line 11, in <module>
    load_entry_point('ESMValTool', 'console_scripts', 'cmorize_obs')()
  File "/home/users/valeriu/esmvaltool_users/esmvaltool/cmorizers/obs/cmorize_obs.py", line 166, in main
    config_user = read_config_user_file(config_file, 'cmorize_obs')
TypeError: read_config_user_file() missing 1 required positional argument: 'options'

@jvegasbsc you need to make that arg optional for console scripts :beer:

jvegreg commented 4 years ago

I am on it, but in the meanwhile you can just past there an empty dict and it should work

valeriupredoi commented 4 years ago

cheers @jvegasbsc - why are we on GitHub on a Saturday? :beach_umbrella:

mattiarighi commented 4 years ago

We are getting there. @bouweandela @valeriupredoi @jvegasbsc it would be good if you could test the recipes assigned to you, unfortunately I missed the data to do this myself.

@earnone @jhardenberg please fix the errors related to your recipes, @katjaweigel can also help with this.

@tomaslovato there are two of your cmorizers which are not working, it would be great if you could fix them.

I still have to test recipe_collins13ipcc.yml, but it requires lots of memory and the DKRZ job scheduler was down in the last few days. I will try again today.

earnone commented 4 years ago

@mattiarighi great. I understood #1752 by @katjaweigel solved the issues with recipe_hyint.yml (#1716), recipe_hyint_extreme_events.yml (#1711) and recipe_extreme_events.yml (#1718). So we are left with recipe_rainfarm.yml (#1715) @jhardenberg . OK?

mattiarighi commented 4 years ago

Correct.

katjaweigel commented 4 years ago

@earnone currently I'm just testing recipe_hyint.yml. Yesterday I had a version which run, but now something went wrong again when I tried to sort my changes into a new branch. Hope I have at least recipe_hyint.yml running again, soon.

jvegreg commented 4 years ago

Regarding my recipes, recipe_seaice_feedback already tested and recipe_seaice_drift is ongoing. I had to cmorize the NSIDC data, as we only have ERA5 and ERA-Interim in the Tier3 of the common data pool. This is intended, @valeriupredoi ? Should I put the data I cmorized there?

jvegreg commented 4 years ago

recipe_seaice_drift is ongoing

Finished ok

valeriupredoi commented 4 years ago

I will spend a good bit of next week making sure all my autoassess shtuff still works fine and is ready for the big launch :rocket: Dealing with some other annoyances, ahem, interesting phenomena in the meantime :grin:

jvegreg commented 4 years ago

@bouweandela, I tried to run the hydrology recipes but I can not find where to get the shapefiles that they require. It is nor in the recipe nor in the documentation. Is there a reason why you are not telling where to get them?

bouweandela commented 4 years ago

@jvegasbsc In principle, the recipes are intended to work with any shapefile describing a river catchment. We recently discussed adding a link to the place where we downloaded them from to the recipe documentation, but this hasn't been done yet. Maybe @nielsdrost can say more about this?

Specifically, copies of the shapefiles mentioned in the recipes can be found here: https://github.com/eWaterCycle/recipes_auxiliary_datasets, but they are just examples.

nielsdrost commented 4 years ago

@jvegasbsc In principle, the recipes are intended to work with any shapefile describing a river catchment. We recently discussed adding a link to the place where we downloaded them from to the recipe documentation, but this hasn't been done yet. Maybe @nielsdrost can say more about this?

Specifically, copies of the shapefiles mentioned in the recipes can be found here: https://github.com/eWaterCycle/recipes_auxiliary_datasets, but they are just examples.

The hydrology recipes should in principle work with any shape. Usually these will come from HydroSHEDS (https://www.hydrosheds.org/), or some custom shape depending on the use-case. We though about including a few example shapes, but could not figure out how to best do this in ESMValTool. Committing them to the repo did not seem to be the best idea, but I also do not really regard them as a "dataset" worth publishing separately. Something for a future ESMValTool workshop discussion, I think.

In the mean time, shall we add a hint on how to get an example shapefile to the recipes?

bouweandela commented 4 years ago

In the mean time, shall we add a hint on how to get an example shapefile to the recipes?

Yes, I think it would be really helpful to put a link to HydroSHEDS in https://github.com/ESMValGroup/ESMValTool/blob/master/doc/sphinx/source/recipes/recipe_hydrology.rst. Maybe we could replace the names of shapefiles in the recipes so their names match a shapefile from that site?

nielsdrost commented 4 years ago

In the mean time, shall we add a hint on how to get an example shapefile to the recipes?

Yes, I think it would be really helpful to put a link to HydroSHEDS in https://github.com/ESMValGroup/ESMValTool/blob/master/doc/sphinx/source/recipes/recipe_hydrology.rst. Maybe we could replace the names of shapefiles in the recipes so their names match a shapefile from that site?

Created a separate issue (#1762)

valeriupredoi commented 4 years ago

tested the ones that had my name against them plus the two validation ones plus the preprocessor derive (note for the last one no OImon data for HadGEM2-A on Jasmin/BADC neither in output1 nor output2, maybe we should use a slightly less obscure dataset?) :beer:

bouweandela commented 4 years ago

preprocessor derive (note for the last one no OImon data for HadGEM2-A on Jasmin/BADC neither in output1 nor output2, maybe we should use a slightly less obscure dataset?)

I changed the mip of the last diagnostic in #1768, so now at least it runs, but just realized that it probably was the wrong way to fix it, because the description is Test sispeed, so probably the variable name was wrong too. A more likely test would be

  diag8:
    description: Test sispeed
    variables:
      sipeed:
        project: CMIP5
        mip: day
        exp: historical
        start_year: 1985
        end_year: 1986
        ensemble: r1i1p1
        derive: true
        force_derivation: false
        additional_datasets:
          - {dataset: MPI-ESM-MR}
    scripts: null

@jvegasbsc I think you originally added this. Is it OK to leave it as is? If not, can you fix it?

jvegreg commented 4 years ago

@jvegasbsc I think you originally added this. Is it OK to leave it as is? If not, can you fix it?

Fixed, waiting for the test to finish on Jasmin...

mattiarighi commented 4 years ago

We are almost there:

@valeriupredoi any news on recipe_autoassess_radiation_rms_cfMon_all.yml?
it looks like we can't fix #1742 without a new ESMValCore release, is there a workaround?

valeriupredoi commented 4 years ago

yes - done, forgot to check the box :grin:

bouweandela commented 4 years ago

it looks like we can't fix #1742 without a new ESMValCore release, is there a workaround?

We could try removing the multi model statistics from the recipes as a workaround, but it looks like @ledm is still working on these recipes in #1331 and #1621, so my impression is that this is not ready for the current release.

mattiarighi commented 4 years ago

That's bad, because it is in the Part II paper. I would suggest to at least remove the multimodel stats if the recipe is running.

ESMValGroup / ESMValTool

Final test for v2.0.0 release #1697