openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference
MIT License
13 stars 11 forks source link

scattered updates for better lost of NWP sources #247

Closed dfulu closed 8 months ago

dfulu commented 8 months ago

Pull Request

Description

A few changes to use a list of zarrs as NWP inputs. Mostly minor changes except perhaps the chunks="auto" in open_mfdataset. I think this is what we want, and from some experiments it makes loading the dataset much faster. I was finding using chunks={} was taking too long to even allow it to complete

Checklist:

codecov[bot] commented 8 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (da3e5a4) 78.58% compared to head (74259ac) 78.60%. Report is 60 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #247 +/- ## ========================================== + Coverage 78.58% 78.60% +0.01% ========================================== Files 129 129 Lines 5674 5693 +19 ========================================== + Hits 4459 4475 +16 - Misses 1215 1218 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.