PVNet2

[dfulu]

Pull Request

Updates for work done under PVNet2

Description

• Updates to use pvnet datapipe from ocf_datapipes
• Write a datamodule which uses DataLoader2 and multiprocessing. Can either construct batches on the fly or load from premade batches
• Some model architecture upgrades (mainly around activation function, batchnorm, skip connections)
• Refactor the model so it is is composable and different model components can be plugged in and out
  • Include encoders module for model components which take 3D satellite/NWP data and encode it down to 1D vector
  • Include linear_networks module for model components which accept multiple 1D feature vectors from different sources and fuse them together into a prediction
• Add variations on the overall model skeleton
  • multimodal - Original model skeleton. Encodes satellite and NWP and fuses them together with GSP history, solar coordinates, and ID embedding.
  • deep_supervision - Adds additional prediction heads to make predictions of GSP future based only on satellite and only on NWP
  • weather residual - Separates data sources so that GSP history, ID and solar coords are used to make initial prediction, and satellite and NWP are used to refine this prediction.
  • nwp_weighting - Completely different architecture where the prediction is a learned weighted mean of the dwsrf channel
• Improved logging during model training

Checklist:

• [ ] My code follows OCF's coding style guidelines
• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

[peterdudfield]

Ive left a few comments, but in general looks really good

I would try to get this merged, and then work on production as a separate branch

[peterdudfield]

are the tests running automatically? Can you fix the pre-commit.ci?

[peterdudfield]

Could you also add a readme.md for the latest experiment you have done, doesnt have to be much text, but I find it really useful to have it near the code,

see example here

[codecov[bot]]

Codecov Report

Merging #6 (1b99771) into main (3c88cc6) will decrease coverage by 25.76%. The diff coverage is 70.42%.

:exclamation: Current head 1b99771 differs from pull request most recent head 43f335d. Consider uploading reports for the commit 43f335d to get more accurate results

@@             Coverage Diff             @@
##             main       #6       +/-   ##
===========================================
- Coverage   92.37%   66.61%   -25.76%     
===========================================
  Files           5       19       +14     
  Lines         354     1333      +979     
===========================================
+ Hits          327      888      +561     
- Misses         27      445      +418     

Impacted Files	Coverage Δ
pvnet/callbacks.py	0.00% <0.00%> (ø)
pvnet/training.py	0.00% <0.00%> (-91.12%)	:arrow_down:
pvnet/utils.py	20.28% <13.55%> (-64.53%)	:arrow_down:
pvnet/models/base_model.py	41.42% <35.29%> (-54.58%)	:arrow_down:
pvnet/models/utils.py	50.00% <50.00%> (ø)
pvnet/optimizers.py	53.33% <53.33%> (ø)
pvnet/data/datamodule.py	75.90% <75.90%> (ø)
pvnet/models/multimodal/deep_supervision.py	79.50% <79.50%> (ø)
pvnet/models/multimodal/basic_blocks.py	88.09% <88.09%> (ø)
pvnet/models/multimodal/encoders/encoders2d.py	88.14% <88.14%> (ø)
... and 9 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more