Optimize functions `mask_landsea()`, `mask_landseaice()` and `calculate_volume()` for lazy input

schlunma commented 3 weeks ago

Description

This PR

uses the newly added chunks for iris.util.broadcast_to_shape whenever possible in our code base to improve lazy computations.
fixes a small bug for mask_landsea (see https://github.com/ESMValGroup/ESMValCore/issues/2514 for details).
makes sure that the functions mask_landsea(), mask_landseaice() and calculate_volume() return lazy data for lazy input, but also realized data if all input is realized.

Closes #2514

Before you get started

[x] ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🧪 The new functionality is relevant and scientifically sound
[x] 🛠 This pull request has a descriptive title and labels
[x] 🛠 Code is written according to the code quality guidelines
[x] 🧪 and 🛠 Documentation is available
[x] 🛠 Unit tests have been added
[x] 🛠 Changes are backward compatible
[x] 🛠 Any changed dependencies have been added or removed correctly
[x] 🛠 The list of authors is up to date
[x] 🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.74%. Comparing base (343368e) to head (325db6d). Report is 2 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2515 +/- ## ========================================== + Coverage 94.70% 94.74% +0.04% ========================================== Files 248 248 Lines 14002 14012 +10 ========================================== + Hits 13260 13276 +16 + Misses 742 736 -6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

schlunma commented 2 weeks ago

The question here is how to treat a mix of input arrays. The common choice seems to be that the output is lazy if any of the input arrays are lazy, e.g. np.ones(2) * da.ones(2) results in a Dask array. This kind of makes sense because it provides the lowest memory footprint and still offers the user full control because they can make all input arrays realized if they want realized output. Would that work for you?

Yes, that makes total sense, and also has the benefit of easier code. I chose to make it only depend on the actual data, not on the ancillary variables. I will change that now 👍

schlunma commented 2 weeks ago

Thanks for the review Bouwe! I implemented all your suggestinos (also for the masking preprocessor here). Could you please have another look?

ESMValGroup / ESMValCore