PV MetNet Results

[jacobbieker]

This is just tracking in a more public way PV MetNet results for site level forecasting. Overall, results so far are probably not worth changing from the random forest model.

The best performing PV MetNet model only took NWP data, and has a MAE of ~11% on average across timesteps up to 30 hours into the future, when only looking at times when generation was above 1% (To cut out night) and on the same set of IDs at the random forest model.

Weights and Bias link is here (WIll need to be signed in to OCF account to see it): https://wandb.ai/openclimatefix/PvMetNet

Detailed Description

PVMetNet was trained with a few different sets of modalities, and not all for the same amount of steps, as especially satellite data is very slow to load. For a similar reason, only the best performing model was benchmarked against the validation set. Overall, since Sun position and topo did not seem to make a huge difference, the one benchmarked was just the NWP-only model. The model was trained to predict generation for each 5 minute interval, including night time. So the training MAEs are artificially low versus models that only predict during day time.

Inputs	Context Size	Pixel Size	Train MAE
NWP	512km	64	0.03583
NWP+Sun	512km	32	0.03516
NWP+Sun	512km	64	0.03937
NWP+Topo+Sat+Sun	128km	64	0.03461
NWP+Topo+Sat+Sun	128km	32	0.05579
Sat+Sun+Topo	128km	16	0.04593

Training loss plots, helpful-flower, blooming-meadow, and unique-grass are the ones with satellite, trained for a lot less time because of the time to create batches was so high:

Validation results:

[jacobbieker]

MAE per forecast horizon. It seems odd it doesn't just keep going up. The overall average is 0.122