[Meta] Datapipes for Training Checklist

[jacobbieker]

This is the steps needed to make Datapipes work for training as well as production, which was covered in #18. As part of that, these functions for training have already been completed:

1. Loading of basic raw data
2. Transform data and normalization

TODO

Loading

• [ ] Load ERA5 NWP
• [x] Load GFS NWP
• [x] Load new RSS imagery
• [ ] Load new Topographic Data (Copernicus topo maps)
• [ ] Load from HuggingFace Datasets

Selecting

• [x] Tests for selecting time periods, including support for t0 offsets
• [x] T0 offset time picker
• [x] T0 picker

Transform

• [x] Output to non-NumpyBatch
• [x] MetNet Preprocessor
• [ ] Data Corruption Datapipes
• [ ] #8
• [ ] #9
• [ ] #10
• [x] #5

[peterdudfield]

Ive added a training pipe lines in https://github.com/openclimatefix/ocf_datapipes/blob/main/ocf_datapipes/training/nwp_pv.py and it seems to work. It only loads PV and a reduce NWP dataset

[jacobbieker]

@dfulu added the offsets in #194 #192