openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

Speed up loading satellite #158

Closed jacobbieker closed 1 year ago

jacobbieker commented 1 year ago

Otherwise HRV ones also become very huge when concatenated across multiple years, and very much slows down opening them up.

Pull Request

Description

Fixes #132

How Has This Been Tested?

Unit tests

[ ] Yes

Checklist:

[ ] My code follows OCF's coding style guidelines
[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked my code and corrected any misspellings