Miscellaneous lazy preprocessor improvements

bouweandela commented 2 months ago

Description

Various improvements to the preprocessor functions related to handling lazy data.

Avoid inserting broadcasted numpy arrays into Dask graph when applying masks
Always specify the target chunks when broadcasting
Avoid huge chunks with derived coordinates https://github.com/SciTools/iris/issues/5457
Ensure correct chunktype for rolling window operations https://github.com/SciTools/iris/issues/5988

Related to #674

Link to documentation:

Before you get started

[x] ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🧪 The new functionality is relevant and scientifically sound
[x] 🛠 This pull request has a descriptive title and labels
[x] 🛠 Code is written according to the code quality guidelines
[x] 🧪 and 🛠 Documentation is available
[x] 🛠 Unit tests have been added
[x] 🛠 Changes are backward compatible
[x] 🛠 The list of authors is up to date
[x] 🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.66%. Comparing base (11e6aa8) to head (9edbc28). Report is 1 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2520 +/- ## ========================================== + Coverage 94.65% 94.66% +0.01% ========================================== Files 251 251 Lines 14287 14296 +9 ========================================== + Hits 13524 13534 +10 + Misses 763 762 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

valeriupredoi commented 1 week ago

@bouweandela this is a lovely PR - did you want to still work on it? It's been a while since it's been sitting out, let's get it in if you done working on it :beer:

bouweandela commented 1 week ago

It's ready for review. I would like to add more tests, but I'm not sure when I'll have time to do that, so maybe it's better to just go ahead.

ESMValGroup / ESMValCore