probabilistic ML

[peterdudfield]

Detailed Description

It would be great to get probabilistic forecasts

Context

• Useful for user to know the uncertainty
• A simplier approach would be to look at the errors over the last week and use this to estimate probabilities, but I think this might be to simple and not capture new weather events

Possible Implementation

• perhaps we could use? https://xgboost.readthedocs.io/en/latest/python/examples/quantile_regression.html
• Report for each horizon how good the quantile regressors are
• for live this should come out as the pd.dataframe here.
• Different code can sort it being saved in the database.

[peterdudfield]

you ok, to add that to production code?

I added this optional columns what how the dataframe should come out here https://github.com/openclimatefix/nowcasting_datamodel/blob/main/nowcasting_datamodel/models/convert.py#L70

[jacobbieker]

Yeah, I can do that

[dantravers]

I agree that they are close enough across horizons to to one multiplier for upper and one for lower. For median, it's also very close - and the standard deviation (across horizons), .009, of the changes is not much less than the % increase (0.023) so not sure adding this changes much.

The only other analysis I would like to try is one I discussed with Jacob, which is by season or month - to flush out if there is a seasonal shape in the factor.

[This doesn't (as we aren't changing the median) change the numbers we get, but I would think about it as a multipliers to: (90% - mean) and (mean - 10%). If we do this we can compare the up and down multipliers, and apply them potentially to other models outputs easily. ]

Nice work!

[dantravers]

Just seeing last post. I think it's fine if you add this to the production code and in parallel run the analysis on the season / month. Getting it into production is the main thing, and we can work out if we need to modify for changes of seasons as a v2.

[jacobbieker]

you ok, to add that to production code?

I added this optional columns what how the dataframe should come out here https://github.com/openclimatefix/nowcasting_datamodel/blob/main/nowcasting_datamodel/models/convert.py#L70

I was thinking of doing the scaling within the https://github.com/openclimatefix/uk-pv-national-xg/blob/661bb0e586f5ebad8bb35cbff485bfbab9ff6759/gradboost_pv/inference/models.py#L483C30-L483C30 so it shouldn't need to be changed in the production code, and stays within the model code, so we update it when we update the model itself.

[peterdudfield]

you ok, to add that to production code? I added this optional columns what how the dataframe should come out here https://github.com/openclimatefix/nowcasting_datamodel/blob/main/nowcasting_datamodel/models/convert.py#L70

I was thinking of doing the scaling within the https://github.com/openclimatefix/uk-pv-national-xg/blob/661bb0e586f5ebad8bb35cbff485bfbab9ff6759/gradboost_pv/inference/models.py#L483C30-L483C30 so it shouldn't need to be changed in the production code, and stays within the model code, so we update it when we update the model itself.

sounds good