Allow use of PV system data from PV Sites database

[dfulu]

Pull Request

Description

This pull request is intended to allow us to use PV data from the pvsites database.

• When loading training data, the capacity of the PV system was previously taken to be the maximum observed output in the timeseries. When loading from the database, the capacity was set to the metadata value. The maximum observed and the metadata value can be quite different, and if we normalise the PV system data by them this will cause us trouble in production.

  This pull request adds new variables to the PV system DataArray. Previously we only had capacity_watt_power. Now we have observed_capacity_watt_power and metadata_capacity_watt_power instead of this. This is more explicit and should help us avoid normalisation mistakes.

  • However, I'm wondering if it might be better to align the names more closely to the GSP capacity names - which are nominal_capacity_mwp and effective_capacity_mwp?

• Added OpenPVFromPVSitesDBIterDataPipe datapipe function (plus helper functions) to load from pvsites database. The current OpenPVFromPVDBIterDataPipe datapipe function can only be used to load from the pv database.

  I think we intend to move away from using the pv database, so I wasn't sure if we would want to replace OpenPVFromPVDBIterDataPipe to only load from pvsites. Currently, I have left the two functions alongside each other.

• Added new tests for OpenPVFromPVSitesDBIterDataPipe

• Added ApplyPVDropoutIterDataPipe to apply dropout specific for PV systems. Independently dropout each system and set a latency for each system.

• Added tests for ApplyPVDropoutIterDataPipe

• Add to PVNet datapipe to include PV system inputs

• Minor cleaning and refactoring

Checklist:

• [x] My code follows OCF's coding style guidelines
• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

[codecov[bot]]

Codecov Report

Merging #226 (2c39650) into main (c7d8da3) will increase coverage by 0.16%. Report is 2 commits behind head on main. The diff coverage is 87.67%.

@@            Coverage Diff             @@
##             main     #226      +/-   ##
==========================================
+ Coverage   79.94%   80.11%   +0.16%     
==========================================
  Files         126      127       +1     
  Lines        5551     5611      +60     
==========================================
+ Hits         4438     4495      +57     
- Misses       1113     1116       +3     

Files	Coverage Δ
ocf_datapipes/config/model.py	86.38% <100.00%> (+0.05%)	:arrow_up:
ocf_datapipes/convert/numpy/batch/pv.py	100.00% <ø> (ø)
ocf_datapipes/load/__init__.py	82.35% <100.00%> (+1.10%)	:arrow_up:
ocf_datapipes/load/pv/database.py	97.05% <100.00%> (+0.84%)	:arrow_up:
ocf_datapipes/load/pv/pv.py	97.16% <100.00%> (-0.13%)	:arrow_down:
...tapipes/select/drop_pv_sys_generating_overnight.py	100.00% <100.00%> (ø)
ocf_datapipes/training/common.py	98.08% <ø> (+2.54%)	:arrow_up:
ocf_datapipes/training/example/nwp_pv.py	89.28% <ø> (ø)
ocf_datapipes/training/example/simple_pv.py	97.14% <ø> (ø)
ocf_datapipes/training/metnet_pv_site.py	80.76% <100.00%> (ø)
... and 10 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more