Make preprocessor lazy

[bouweandela]

Overview issue with laziness status of preprocessor functions:

Checked means lazy, unchecked means not lazy or partially lazy, a question mark behind the preprocessor name means that it is unknown whether this preprocessor function is lazy or not.

• [ ] fix_file - copies files on disk #2454
• [x] load
• [x] fix_metadata - depends on model/variable
• [x] fix_data - depends on model/variable
• [x] cmor_check_metadata
• [x] cmor_check_data
• [x] concatenate
• [ ] derive - lazy for most variables, see #2451
• [x] extract_time
• [x] extract_season
• [x] extract_month
• [x] resample_hours
• [x] resample_time
• [x] extract_levels
• [x] weighting_landsea_fraction
• [x] mask_landsea
• [x] mask_glaciated
• [x] mask_landseaice
• [ ] regrid - yes in most cases, for 1D coordinates with the built-in schemes and for other cases with the generic schemes provided by iris-esmf-regrid. See #2405 for more information on 2D latitude and longitude coordinates.
• [ ] extract_coordinate_points - No, see #2446
• [ ] extract_point - No, see #2446
• [ ] extract_location - No, see #2446
• [x] mask_multimodel - yes, but reads all data twice
• [x] mask_fillvalues #2340
• [x] mask_above_threshold
• [x] mask_below_threshold
• [x] mask_inside_range
• [x] mask_outside_range
• [x] clip
• [ ] rolling_window_statistics - not yet, but will be in iris 3.10, see https://github.com/SciTools/iris/pull/5795
• [x] extract_region
• [x] extract_shape
• [x] extract_volume
• [ ] extract_trajectory - no, underlying iris.analysis.trajectory.interpolate function is not lazy
• [x] extract_transect
• [x] detrend - no non-lazy version available
• [x] extract_named_regions
• [x] axis_statistics
• [x] depth_integration
• [x] area_statistics
• [x] volume_statistics - #647, #1498
• [x] local_solar_time
• [x] amplitude
• [x] zonal_statistics
• [x] meridional_statistics
• [x] accumulate_coordinate
• [x] hourly_statistics
• [x] daily_statistics
• [x] monthly_statistics
• [x] seasonal_statistics
• [x] annual_statistics
• [x] decadal_statistics
• [x] climate_statistics
• [x] anomalies
• [x] regrid_time
• [ ] timeseries_filter - not yet, but will be in iris 3.10, see https://github.com/SciTools/iris/pull/5795
• [x] linear_trend
• [x] linear_trend_stderr
• [x] convert_units
• [x] ensemble_statistics
• [x] multi_model_statistics
• [x] bias

Note that *_statistics preprocessor functions are lazy except for median, workaround is to use operator: percentile; percent: 50.

It would be great if we could make more preprocessor functions lazy. The laziness status should also be indicated in the docstrings.

Related to #51

[valeriupredoi]

good call!! I reckon this is a good first serious feature for v2.1 :beer:

[bouweandela]

To get an idea of the priority of the preprocessor functions, here is a rough count of the number of recipes in the ESMValTool that they are used in:

regrid 60
extract_region 30
derive 26
extract_levels 24
mask_landsea 21
area_statistics 20
climate_statistics 18
multi_model_statistics 16
mask_fillvalues 15
annual_statistics 8
convert_units 8
weighting_landsea_fraction 7
anomalies 7
extract_point 6
zonal_statistics 5
extract_time 4
extract_shape 4
amplitude 4
extract_season 3
detrend 3
extract_month 2
extract_transect 2
depth_integration 2
volume_statistics 2
mask_landseaice 1
extract_volume 1
extract_trajectory 1
extract_named_regions 1
meridional_statistics 1
daily_statistics 1
decadal_statistics 1

[Peter9192]

Two points related to this:

• It would be helpful to implement a lazy-in / lazy-out test or check for each preprocessor (we can use xfail on those that we know are not lazy yet).
• We should think about when and where to rechunk the data. Now I'm inclined to do ad-hoc rechunking whenever I need it, but this may not be the most efficient approach in the long run. See e.g. https://github.com/ESMValGroup/ESMValCore/pull/968#issuecomment-796817147

[bouweandela]

Now I'm inclined to do ad-hoc rechunking whenever I need it

I think it makes sense to do that, because in most cases there is no need to rechunk. I would expect this is needed only for preprocessor functions that dramatically increase chunk size, and even then it might be best to try and leave that to iris.

[Peter9192]

Some things we ran into:

• We start to see that the timing of log messages becomes confusing for lazy evaluation.
• We should consider the interplay between the multiprocessing on tasks and the multithreading by dask.
• Maybe we should consider setting up some global dask configuration, or a way to tune easily.

[remi-kazeroni]

The recipe_collins13ipcc is one of the most demanding memory recipe that we have in ESMValTool, see log file for a test run for the v2.5 release. Would it be a case that would benefit from preprocessor laziness?

[bouweandela]

Yes, it would benefit. Some progress has been made recently by @zklaus on the lazy regridding for ocean data (though I think this is currently not yet enabled by default, you need to make changes to the recipe in order to use the new functionality and install an extra package manually). Lazy vertical interpolation would be a good next candidate to tackle.