Closed k-a-webb closed 1 month ago
@malininae
Thanks for reporting the issue! This happens because the climate_statistics and extract_levels preprocessor functions are now lazy, but the ESMPy based regridding preprocessor is not. Therefore it was loading the data from disk and recomputing the input to the regridding multiple times. It should be fixed by https://github.com/ESMValGroup/ESMValCore/pull/2418.
(Note: ESMValTool install main branch fails with missing author message -- despite inclusion of authors.)
It looks like you did not install ESMValTool, but only created the conda environment with its dependencies. If you run pip install -e .
in the directory where you checked out ESMValTool it should work as expected.
Excellent! Thanks for sorting this out :D
For a single CanESM5 dataset of 30 years, it takes ~4min wall time to run a simple recipe in ESMValTool v2.5.0, but >1h in v2.10.0 and v2.11 (main branch). There is a significant increase in run time in the regridding step.
The test recipes involves the following preprocessors (and no diagnostic scripts):
In this case, the regridding is done via
esmpy_regrid
(alias of_regrid_esmpy.regrid
) as it passes_attempt_irregular_regridding
check)It takes longer to both build the target grid (
regridder = build_regridder(src_rep, dst_rep, method)
), and regrid the data (res = map_slices(src, regridder, src_rep, dst_rep)
).main_log_debug.py
files for the various runs of the same recipe in different environments, each with a different installation of ESMValTool: main_log_debug-esmvaltool.txt main_log_debug--ESMValToolv2.5.0.txt main_log_debug-EVTmaindev.txt (same as main_log_debug-EVTmain.txt, as expected)(Note: ESMValTool install main branch fails with missing author message -- despite inclusion of authors.)
Installation details:
ESMValTool v2.10.0 was installed via
ESMValTool main branch installed via
as well as the development version,
Note: To install ESMValTool v2.5.0 the following modifications to the install instructions was required:
Following the basic instructions for installing ESMValTool without the above modifications lead to package version issues with both shapely and esmpy/ESMF
Environment files (
conda list > environment_<env>.yml
) files are also attached. environment__EVTmain.txt environment__esmvaltool.txt environment__EVTv2.5r.txt environment__EVTmaindev.txt