Probablistic v0

[peterdudfield]

Detailed Description

Should we do something really simple for the probabilistic forecast first of all.

• Calculate the average error beween forecast and pvlive update estimate for every model, time horizon and time of day (this can be done in this repo). This can be stored in the db
• when loading the adjuster, also load this value std values, and create 10 and 90 plevels

These errors bars are basically the error over the last 7 days for the models. These are naturally then less good as the dont include what the NWP are doing right now.

Context

• Wp4 coming to a close, so should get a probalistic forecast out there
• v1 probablist can still be with National_xg

Possible Implementation

1. add std metric, like the ME metric for the adjuster
2. load these values when loading the ME in the adjuster, and create plevels, assuming normal distribution.

I think this would take 2 days of work

[JackKelly]

This is an interesting idea :slightly_smiling_face: - and, in general, I really like the idea of trying to do something simple.

However, AFAICT, the uncertainty is mostly dependent on the weather regime (e.g. is it a clear day? or is it cloudy). I'm not sure if the uncertainty is particularly dependent on time of day etc. (other than the trivial relationship that uncertainty scales with PV output). So, yeah, this approach will produce some numbers. But will those numbers be informative? Or will it just always say "maximum uncertainty is a midday, and the furthest forecast horizon"?

Honestly, getting a NN to output a probability distribution (MDN) is pretty simple (maybe an additional 5 to 10 lines of code?). What do you think, @dfulu ?

I'm also a bit cautious about getting a "placeholder" into production. It's all to easy to then de-prioritise getting a fully functioning system in place ("well, it works as-is, so we should prioritise other work.")

That said, I don't want to completely shoot down this idea - I'm sure it'll produce numbers that, at least at first glance, look informative. And would allow us to test the system end-to-end.

[dantravers]

I think a key driver is we said we would do probabilistic in WP4, a payment milestone.

I think there are three options:

1. Very simple from Peter
2. Do ML method on PVNet 1 / 2. This would give probabilistic for 8 hours, which would be enough in my mind for now at least.
3. What Jacob is doing on national_xg - and we could possible multipley them by some factor to ensure they capture the variance of the actual error distribution better.

[peterdudfield]

Perhaps we should do all or 1. 2. and 3. This would de risk the situation.

I totally get what you mean @JackKelly we shouldnt put un-useful stuff into production. On the flip side, getting something out there with the user, I think it is very useful, even if its just internal, and we can turn it on/off on production side.

[jacobbieker]

Yeah, I think the very simple one might be good. I would be more inclined to scaling the output from national XG model variance to match more what the actual error is, its fairly consistently low by the same amount throughout the CV, so rescaling it should be fairly simple, and then includes some of the effect from the NWP at least? I think the best method would be with probabilistic with PVNet in the end, and extending PVNet to the 36 hours, but that will take longer.

[JackKelly]

Rescaling xg_boost variance sounds good. (how much effort would that take to put into production?)

The downside of implementating all three options is that it guarantees that work will be wasted, and the team is fairly busy. Although I take the point that implemting all three would reduce risk (at the cost of being inefficient wrt the team's time)

[peterdudfield]

Rescaling xg_boost variance sounds good. (how much effort would that take to put into production?)

The downside of implementating all three options is that it guarantees that work will be wasted, and the team is fairly busy. Although I take the point that implemting all three would reduce risk (at the cost of being inefficient wrt the team's time)

I think it would be pretty hard to re-scale xg_boost dpending on the effect of NWP. It would involve having lots of NWP data in the cloud in order to learn this re scallings. Other option is to learn this rescalling offline, and that would be a model that would try to learn the variance of a model with inputs as NWP. This feels like what we are doing with XG boost already in https://github.com/openclimatefix/uk-pv-national-xg/issues/66

Yea there is that downside, of some code being wasted. Although if we have probablistic on National_xg and probalistc on pvnet v2 then they could be both used.

[dantravers]

I think this one:

scaling the output from national XG model variance to match more what the actual error is

Would be just a constant scaling factor, as Jacob's analysis of the % covered by his out of the box probabilistic measures shows a pretty constant % is covered across times of day and years. I definitely don't think we should be changing the scaling based on weather for a MVP.

[dantravers]

Maybe it would be worth checking if the % is constant across seasons.

[peterdudfield]

Yea sorry I miss understood something, the method of scalling the plevels in the National_xg model is probably a good way to go

[dfulu]

I'm happy with whatever option we go with. I do agree with @JackKelly's idea that to add probabilistic to PVNet seems like it should be fairly straight forward. It would be a little more than 5-10 lines of code because of logging and extra bit of code, but seems like it should be less than a week's work to write up and train

[dantravers]

So approach we have as I understand is: 1) Jacob to implement the scaling on national_xg probabilistic to give first production version. Might have to think about if we can take the 90-10 envelope and apply it to the first few hours when the national_xg is blended with PVNet. but tbd. This will satisfy our milestone for NG. 2) James invesitigating PVNet 2 probabilistic.

[jacobbieker]

Yes, that's what I believe we are going with.

[JackKelly]

So approach we have as I understand is...

SGTM!

[JackKelly]

A question for the clever mathsy people on the team :slightly_smiling_face: ...

Do we know how we'd blend a probabilistic xg_boost prediction with a probabilistic PVNet2 prediction?!

[peterdudfield]

A question for the clever mathsy people on the team 🙂 ...

Do we know how we'd blend a probabilistic xg_boost prediction with a probabilistic PVNet2 prediction?!

A bit of a tough one, I think the best way to do it is with weights.

So the same withe mean we blend the "plevel_50" with a series of weights to make it smooth. There are current set here. These are easy if they are the same "plevels", I think it gets a bit hard if they are not.

Oh and its good to normalize the plevels around plevel 50 when blending, to make sure they are all centered around the plevel_50

[peterdudfield]

I'm gona close this for the moment. This is v0 done. Of course more dicsuiion to have about PVnet making probablistic levels