openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

Add converting batches to NetCDF and saving them out #238

Closed jacobbieker closed 9 months ago

jacobbieker commented 9 months ago

Pull Request

Description

This PR adds support for converting multiple input dataarrays/datasets into a single Dataset to save it out as NetCDF. This is to help with #231 and make batches more interpretable on disk .

Fixes #237

How Has This Been Tested?

Unit tests

Checklist:

codecov[bot] commented 9 months ago

Codecov Report

Attention: 47 lines in your changes are missing coverage. Please review.

Comparison is base (2d67722) 78.60% compared to head (62ea906) 79.05%. Report is 7 commits behind head on main.

Files Patch % Lines
ocf_datapipes/training/common.py 75.83% 36 Missing :warning:
ocf_datapipes/training/windnet.py 92.80% 10 Missing :warning:
ocf_datapipes/utils/utils.py 97.14% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #238 +/- ## ========================================== + Coverage 78.60% 79.05% +0.45% ========================================== Files 129 130 +1 Lines 5693 5868 +175 ========================================== + Hits 4475 4639 +164 - Misses 1218 1229 +11 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jacobbieker commented 9 months ago

This has the start of windnet forecasting, but mostly is just adding how to combine and uncombine the output from PVNet datapipe to save to NetCDF, and load it from that.