Any idea whether these 2-14 day forecasts are available from NG/ESO?

[deltamelter]

Maybe this is somehow guessitimated/interpolated from single value per day which is all I can find, but this site suggests to be taking this from NGC/ESO API Would be great to have some days ahead in HA, (although 14 days seems to be too far to forecast)

https://emoncms.org/ukgrid/graph

[JRascagneres]

Currently I just do a couple days ahead.... I'm not entirely sure why now...

I had been reading from Elexon who currently provide most of the data to the integration who provide this statement:

National Grid ESO uses its wind power forecasting tool to produce hourly forecast for period from 20:00 (GMT) on the current day (D) to 20:00 (GMT) (D+2).

I had assumed that meant the data wasn't available. It does appear that NG ESO has the data though. It may be viable to add some new sensors which are 'long-term' look ahead sensors.

[JRascagneres]

Sensors DO have a size limit of 16384 bytes though which 14 days may exceed so I'll have to figure it out

[deltamelter]

Sensors DO have a size limit of 16384 bytes though which 14 days may exceed so I'll have to figure it out

Well it's all pure speculation beyond 3-5 days anyway. How about "graded" forecasts? eg break the forecasts into multiple sensors a) D & D+1 (most "accurate") b) 2 to 5or7 (variable but good indication of the week ahead) c) 5or7 to 14 (pure guess, but nice rough guide) this doesn't need to be any more than 4-5 readings per day (eg 2am, 7am, 12pm, 5pm, 10pm)

Also National Demand Forecast seems to available up to 14 days and along with renewables forecasts helps to predict what the wholesale price might do in the week(s) ahead

[JRascagneres]

I'll have a think. I think I'll keep the existing stuff and this will be additional. Splitting out to the next 7 days and then the following 7 days makes sense. I can add the demand forecasts too. Will need to see how much data I can add before it hits that byte limit.

[JRascagneres]

Just want to say I haven't forgotten about this. I've just been busy

[deltamelter]

Looking at these number a lot this week 😬 with no good news from the longer term wind forecast

[JRascagneres]

@deltamelter Hey. I added a few new entities. See the README. Currently its only available as pre-release. You need to tick 'beta' in HACS to get it available - https://hacs.xyz/docs/faq/beta/ Let me know how it is?

[deltamelter]

Thanks. I appreciate you finding the time for this.

I'm not using HACS so manual drop in is fine. Did you manage to find national demand, embedded wind and solar forecast for 14 days?

Would it be possible, with the reduced value frequency, to do now to 14d and now to 7d. Otherwise for full 7 days, I have to combine the entity arrays and that is hard/impossible to do in apex charts. using multiple entities to show effectively the same line messes with scale and stacking in apexcharts too. (have similar issues with prev, current and next events in OE)

[JRascagneres]

@deltamelter I'll see what I can do. I didn't get around to checking the others but its possible I could find them. I could probably do a 3-hourly 7 day and a 3-hourly 14 day and drop the 3 day one? I'll have to poke around. Might be able to do a full 14 day one with the lower accuracy.

[deltamelter]

@deltamelter I'll see what I can do. I didn't get around to checking the others but its possible I could find them. I could probably do a 3-hourly 7 day and a 3-hourly 14 day and drop the 3 day one? I'll have to poke around. Might be able to do a full 14 day one with the lower accuracy.

half- hour accuracy is great, so I'd prefer to keep that. doesn't seem worth keeping both 7d and 14d at the same accuracy, so if you can manage full 14 days, then that and 3d of 30min would be perfect.

[JRascagneres]

@deltamelter I'll see what I can do. I didn't get around to checking the others but its possible I could find them. I could probably do a 3-hourly 7 day and a 3-hourly 14 day and drop the 3 day one? I'll have to poke around. Might be able to do a full 14 day one with the lower accuracy.

half- hour accuracy is great, so I'd prefer to keep that. doesn't seem worth keeping both 7d and 14d at the same accuracy, so if you can manage full 14 days, then that and 3d of 30min would be perfect.

I'll see what I can do. Yeah 30 minute is the default but it was too much data. If I recall the 7 day of 30 minute was nearly 2x the limit. I reckon I can probably do 2 hourly or 3 hourly for 14 days. I'll have a play.

[JRascagneres]

Hi. So I kept the new 3 day one the same. I removed the 7 day one. The new 14 day one goes from now - 14 days and is two-hourly

[JRascagneres]

I have not yet added the others. I will look into it. I should be able to get that data, may be able to do it tomorrow.

[JRascagneres]

Okay beta release v0.0.32 has embedded solar and wind for fourteen days. Its two-hourly. Still looking for the demand data.

[deltamelter]

Great, thank you!! could you add the same 3 day accuracy scales for the other entities? The 14 day is a bit too "rough" for 2-3day view. Also trying to figure out how to look at history, just the last 8-24 hours. but these new ESO data sources seem to all start at "now" Stacking in apex is quite annoying and difficult to do with multiple data sources, the green line across the middle of this is 3-day wind which is incorrectly stacking on the original 2 day wind which includes history. The wind and solar is also flipped as a first attempt at visualising the upcoming effect on Agile Rates. I want to incorporate demand into this too, but also to roughly forecast agile price for the week ahead.

[JRascagneres]

Yeah unfortunately the national grid APIs only provide data from now so history is difficult. I'll look at the rest later on

[JRascagneres]

@deltamelter Added the three day embedded solar + wind to https://github.com/JRascagneres/HA-NationalGrid/releases/tag/v0.0.33

Still need to look at the demand. Super interested in your agile prediction algorithm once you figure that out too!

[JRascagneres]

@deltamelter Hey. I've added the day ahead demand forecast. Not got around to the 14-day one yet, however, this should be sufficient to plug in most of the data and work on your price forecasting https://github.com/JRascagneres/HA-NationalGrid/releases/tag/v0.0.34

[deltamelter]

Day ahead is not quite far enough ahead of what we already know from the wholesale auction price for the next day. While 14 days would be much rougher guess, it is what is used by the generators for spin up planning.

This API link gives NationalDemand for full 3 days at half-hour resolution: https://api.nationalgrideso.com/api/3/action/datastore_search?resource_id=7c0411cd-2714-4bb5-a408-adb065edf34d While this SQL query gives up to 14 days (also half-hour resolution): https://api.nationalgrideso.com/api/3/action/datastore_search_sql?sql=SELECT%20*%20FROM%20%20%227c0411cd-2714-4bb5-a408-adb065edf34d%22%20ORDER%20BY%20%22_id%22%20ASC

[deltamelter]

We already know tomorrow's prices, and demand for the day ahead is looking low, but just stops as it's getting interesting :grinning:

[JRascagneres]

@deltamelter Hey. Thats neat. I'll get the demand implemented later on. Is the same as the other data good enough? Like a now-three days and a now-14 days with 30 minute and 2 hourly resolution respectively? I can get that done tonight I think.

I'll use this - https://api.nationalgrideso.com/api/3/action/datastore_search?resource_id=7c0411cd-2714-4bb5-a408-adb065edf34d&limit=1000

That is 30 minute resolution for the 14 days but I won't be able to easily fit that in the HA output. Will that work?

Also... In your graph, whats the different things meaning, trying to get data from it :P How was the price 'guesser' before the data came out?

[deltamelter]

Yes that would be great in same resolutions as the wind and solar forecasts. It was interesting to see every half hour for 14-day demand forecast, but would never use this many data points so happy to have that at lower resolution.

On the graph, the right-side scale is upside down starting at minimum of 100MW, though I would usually only show minimum of 6000-10000MW as this is where it seems most likely to impact wholesale prices. Stacked and shaded areas are wind, embedded wind and solar. red line is demand (not stacked).

Left-side scale is the agile prices , first based on EPEX 60 as 09:30 and gives a good indication of the overlaid EPEX 30 which is published at 15:45. This graph is not using Octopus API which seems to be published later and later.

[JRascagneres]

I'll try to get that demand stuff sorted tonight for you. Super interested if you're able to do whacky stuff and predict prices :D

[JRascagneres]

@deltamelter Hopefully nearly there now. Added the three and fourteen day demand forecasts - https://github.com/JRascagneres/HA-NationalGrid/releases/tag/v0.0.36

[JRascagneres]

@deltamelter How has it been? Got the prediction and / or data needed? (I think I see you pop up in Octopus Reddit)

[deltamelter]

still working on the predictions beyond 2-3 days but get good visual indication from graphing the data for now

[JRascagneres]

Sweet. Might close this out then and release the full version (bring out of pre-release). Can you share any prediction once you get it? Super interested. Okay if not.