Closed dfulu closed 9 months ago
@peterdudfield and @jacobbieker, any thoughts?
Yea i agree, start with 2, and move on to 3 later.
Following on from your analysis earlier, might also be interesting to see if the accuracy per GSP improved by using PVlive intraday values
Yeah, I also agree, 2 first, then 3
Is there now a 4th option, which is to use PVlive intraday in training, now we have that from Sheffield Solar? I'd porbably go
Yes that is an option now with the backtest data we have. I think I'd prioritise doing 4 and 2 together. 4 should be reasonably quick to do, its essentially just retraining a model after modifying the batches I already have saved out. 2 will be the same amount of work regardless of whether I do 4 or not
I've been doing some sanity checks on the backtest of PVNet data - particularly looking at the national values.
Unfortunately the backtest we got from sheffield solar doesn't match the values in the database that we've been scraping from them in production. The comparison between the PVLive initial estimates we have in the database (i.e. the values we have at production time) and the backtest values are shown in the figure below. The backtest values are biased to be too high.
The figure below is trying to answer "are the backtest values closer to the initial estimates we have in production than the updated values?". Based on the graph below, I think the answer is yes, but probably not close enough to solve the issue in #53.
I think this likely means that we cannot do 4 without recreating the same issue as is #53. Similarly, 2 may not work for the same reasons.
I've been doing some sanity checks on the backtest of PVNet data - particularly looking at the national values.
Unfortunately the backtest we got from sheffield solar doesn't match the values in the database that we've been scraping from them in production. The comparison between the PVLive initial estimates we have in the database (i.e. the values we have at production time) and the backtest values are shown in the figure below. The backtest values are biased to be too high.
The figure below is trying to answer "are the backtest values closer to the initial estimates we have in production than the updated values?". Based on the graph below, I think the answer is yes, but probably not close enough to solve the issue in #53.
I think this likely means that we cannot do 4 without recreating the same issue as is #53. Similarly, 2 may not work for the same reasons.
This may be because the backtest is run on the 1000 systems, but in relately perhaps only 700 sites are reporting live, and this could make a different.
hmm, perhaps the best way is to just use some PV Passive data, then we cut out the middle algorithm (and errors), this perhaps it simplier than using mulite GSP PVlive datasources
Closing now as we do not plan to use PVLive inputs any longer and have already been experimenting with Passiv
We want to use some historic measurements to improve the accuracy of the model, particularly at short time horizons. However, using intra-day inputs to a model trained on updated PVLive decreases the accuracy of the model beyond ~1-hour time horizon (see #53). This is down to the differences between the initial estimate (intra-day) and the updated estimate. It's also important to note that after training PVNet we need to train the summation model so as to have probabilistic national prediction.
Assuming that we cannot get a significant backtest of GSP level PVLive intra-day estimates. Some solutions could be:
The easiest solution here would be 2. After solving issues with the Passiv data, we could combine 2 and 3