openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

new gfs loader #338

Closed AUdaltsova closed 1 week ago

AUdaltsova commented 1 week ago

Pull Request

Description

New version of GFS loader added:

New GFS data has step 0 missing for irradiance (variables dswrf, dlwrf), currently defaulting to NaN.
Added a commented-out bit that extrapolates them in the loader
Added a warning if using irradiance
Added test data for gfs
Added standard test for gfs loader

How Has This Been Tested?

ran small batch creation on leonardo with and without irradiance variables

Checklist:

[X] My code follows OCF's coding style guidelines
[X] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[X] I have added tests that prove my fix is effective or that my feature works
[X] I have checked my code and corrected any misspellings

AUdaltsova commented 1 week ago

hang on I'll add mean and std here as well