Avoid loading entire files into memory when downloading from ESGF

bouweandela commented 4 months ago

Description

Due to https://github.com/psf/requests/issues/5536, downloading files from ESGF loaded entire files into memory, leading to potentially high memory use when downloading large files. This pull request avoids the issue.

Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🧪 The new functionality is relevant and scientifically sound
[x] 🛠 This pull request has a descriptive title and labels
[x] 🛠 Code is written according to the code quality guidelines
[x] 🛠 Unit tests have been added
[x] 🛠 Changes are backward compatible projects/ESMValCore/en/latest/contributing.html#dependencies) correctly
[x] 🛠 The list of authors is up to date
[x] 🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.51%. Comparing base (2fc8761) to head (04d522e). Report is 41 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2434 +/- ## ======================================= Coverage 94.51% 94.51% ======================================= Files 246 246 Lines 14023 14024 +1 ======================================= + Hits 13254 13255 +1 Misses 769 769 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

bouweandela commented 4 months ago

@chrisbillowsMO if it is still possible, this bugfix would also be nice to have in the release.

chrisbillowsMO commented 4 months ago

@chrisbillowsMO if it is still possible, this bugfix would also be nice to have in the release.

@bouweandela Shouldn't be a problem. Have added to our checklist.

valeriupredoi commented 4 months ago

Many thanks @chrisbillowsMO - merge at will 🍺

ESMValGroup / ESMValCore