PVnet backtest analysis

[peterdudfield]

Detailed Description

We have run a backtest for PVNet and we shoudl do some analysis on this

Context

• good to be able to send to people, to show how good our forecast is

Possible Implementation

Merge national all together from each init forecast

Metrics:

• MAE
• RMSE
• R2
• p10,p25,p75,p90 absolute errors using pinball score
• MBE?
• Ramp rate

We should compare A: PVnet to PVLive update for the whole of 2022 B:

1. PVnet to PVLive update for the whole of 2022-07-01 to 2023-01-01
2. PVLive initial to PVLive update for the whole of 2022-07-01 to 2023-01-01

[peterdudfield]

We can add comments on here @zakwatts

[zakwatts]

Outline of plan:

Step 1: Load in all of the 3 data sources.

Step 2: Clean the data so I only have the exact data that i need

Step 3: Merged all of the data together into a single pandas dataframe.

Step 4: Create a chart plotting system to specify a specific day or number of days to check if the data looks good. Also check for NaNs and any missing data. Sort appropriately. Make sure that I am using the OCF colour scheme.

Step 5: Create a way to subset the data into respective hourly, weekly and monthly formats. This could be done by having distinct datasets for each resolution. (it wouldn’t make sense to pre-process the data each time). [one ds for hourly, for this create a weekly and monthly dataset] A function could be made that creates that chart, which can then be called using each of the datasets for each of the resolutions. Def make_chart: CHART make_chart(ds_hour) Would also want to make sure I could do this for each month/week. Def hour averaged for each week (graph_type, ds): For in weeks… Filter data for that week graph_type(ds) (Alternatively just use a heat map approach and dont make graphs for each week as might not be super insightful…)

Step 6: Create plots comparing metrics

Step 7: Create plots comparing relative differences based on those metrics

Step 8: Potentially create heatmaps of the different plots ((x)hours vs (y)weeks/months with MAE/%_diff_in_MAE shown as colour intensity.

Step 9: Bring it all together in a report

[peterdudfield]

As a step 0, could you share the location of the merged forecasts. Perhaps put it back in the gcs dir

[peterdudfield]

Extra

• look at different forecast horizon. plot MAE vs forecast horizon to get an idea of how good it.
• Look at the distribution of daily MAE, look att std - maybe T test, to check its better then PVlive intraday
• Run Pinball loss on each quantile forecast for PVnet
• bonus: National_xg -

[dantravers]

I'd definitely like to see step 8, as it's interesting, and they are looking for seasonal / time-of-day analysis. For probabilistic, I would add just a simple count of exceedences. General principle - let's play around a bit, but then narrow down on the metrics we wish to see repeated when we do this analysis again, as I'm 100% sure we will want to re-run this in another few months as we improve PVNet2. I would put National_xg as out of scope for now, or let's discuss before embarking on this.

[peterdudfield]

I'd definitely like to see step 8, as it's interesting, and they are looking for seasonal / time-of-day analysis. For probabilistic, I would add just a simple count of exceedences. General principle - let's play around a bit, but then narrow down on the metrics we wish to see repeated when we do this analysis again, as I'm 100% sure we will want to re-run this in another few months as we improve PVNet2. I would put National_xg as out of scope for now, or let's discuss before embarking on this.

For probablistic, count exceedences is useful, but pinball gives a understanding of good their error bars are too. You can have the number of exceedences be correct but have very wide error bars. pinball loss is pretty industry standard

Yea, happy to delay National_xg - or discuss beforehand

[dantravers]

Yep, exceedence and pinball, agreed.

[dfulu]

I'm going to close this because I think it has been completed