Refactor

[dfulu]

Pull Request

Description

A refactor and spring clean of datapipes

Changes are:

• Remove 2 unused functions from experimental
• Remove config loading duplication
• Rename DropGSP to SelectGSPIDs since that describes the function better
• Remove OffsetT0IterDataPipe since its functionality is duplicated in SelectDropoutTimeIterDataPipe
• Remove select_live_t0_time and select_live_time_slice as they were unused in either PVNet or xgnational production
• Remove unused TrimDatesWithInsufficentData function
• Remove 5/15 minute satellite overloading logic. This should be in production app(s) e.g. openclimatefix/pvnet_app#32
• Remove function for loading PV data from the old (now defunct) database
• Remove duplicate functions for slicing NWPs for forecast time t0
• Remove create_pv_history_image since we already have create_pv_image which is better
• Rename and move some files
  • This includes establishing a preffix convention in ocf_datapipes/select:
    • select_ is reserved for sample level selection, like selecting a window in time or space for a sample
    • filter_ is reserved for dataset level selection, like slicing out the train/test period, selecting channels, or systems by ID
    • pick_ is preserved for functions which take a dataset and yield locations and/or t0 times
  • Remove the number_of_locations function and the part of pick_to_times which was intended to use it. If we want all space and time locations we should use pick_locs_and_t0s instead.
  • Remove remove_northern_gsp. This was to remove Shetland, and we in general can predict for this region now
• Remove function save_to_time to add fourier t0, which isn't used and probably doesn't make sense in fourier space
• Remove satellite float [0-1023] to int8 conversion. This function was unused
• Remove unused keep_dawn_dusk_hours from config model

Checklist:

• [x] My code follows OCF's coding style guidelines
• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

[codecov[bot]]

Codecov Report

Attention: 42 lines in your changes are missing coverage. Please review.

Comparison is base (e3e8de3) 78.12% compared to head (73c9ab0) 77.51%. Report is 2 commits behind head on main.

Files	Patch %	Lines
ocf_datapipes/training/metnet/metnet_national.py	21.42%	11 Missing :warning:
...cf_datapipes/training/metnet/metnet_pv_national.py	21.42%	11 Missing :warning:
...f_datapipes/training/metnet/metnet_gsp_national.py	27.27%	8 Missing :warning:
ocf_datapipes/training/common.py	86.95%	3 Missing :warning:
ocf_datapipes/training/metnet/metnet_pv_site.py	50.00%	3 Missing :warning:
ocf_datapipes/training/pseudo_irradience.py	50.00%	3 Missing :warning:
ocf_datapipes/load/satellite.py	71.42%	2 Missing :warning:
ocf_datapipes/select/pick_t0_times.py	94.44%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #256 +/- ## ========================================== - Coverage 78.12% 77.51% -0.61% ========================================== Files 132 118 -14 Lines 5792 5364 -428 ========================================== - Hits 4525 4158 -367 + Misses 1267 1206 -61 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[dfulu]

closes #252