Per-site forecast today

[gcoan]

Is your feature request related to a problem? Please describe. On sunny or reasonably sunny days I consistently fail to achieve the forecast PV50 generation, I usually get somewhere between the PV10 and PV50, but today I didn't even generate the PV10 figure:

Now I know this is just a forecast but I'm fairly sure that the major contributory factor to this is my inverters shutting off because I'm exporting so much that I'm increasing the grid power too much at peak generation times.

Power generation (in kW) follows the forecast power (in W) until it gets to about 8kW when shutoff's occur:

Forecast Power Now has the per-site data so I can graph each site forecast to actual, but what I can't do is quantify the total impact of this because I don't have the per-site forecast generation, I only have the total across all sites.

Describe the solution you'd like Add per-site forecast figures to pv_forecast_today, _tomorrow, _d3. [I'd probably like the PV10, not bothered about PV90 or D4+ TBH, but I'm sure you will want to add them all for symmetry]

Describe alternatives you've considered Direct API call to solcast? Something funky working out the area under the power now graph?

[autoSteve]

Just the day roll-up, or the detailedForecast too?

[gcoan]

Just the single total daily figure is all I am looking for then I have a figure I can compare to the actual generated PV per site to quantify what I am missing out on. The detailedForecast is vaguely interesting but I don't think I'd use it and its too much attributes. The per site PV power now gives me something to compare the actual PV output to

[autoSteve]

Check out the branch Daily-forecasts-site-breakdown, @gcoan.

[autoSteve]

I've done a thing, @gcoan, which might never see the light of day.

There are many files changed, including translations, which has got "stuff up" written all over it because Aussie relying on Google translate.

This adds a config upgrade to v8, adding options for whether site and estimate detail are included as attributes. The options upgrade should be "compatible" with earlier versions. (How's your Urdu and Slovak?)

My (evil laugh) plan is to evolve this to optionally include the detailed forecasts breakdowns as "opt-in" attributes, not defaulting to True, and as separate options.

[gcoan]

That sounds a good thing @autoSteve , making the site and estimates in attributes a configurable option so people don't get cluttered with too much unnecessary stuff.

There's also a risk (and we've seen it with Predbat) that if you stuff too much in the attributes then you start getting warnings in the HA log from recorder that it can't store more than 16k of attribute values - despite the fact that it doesn't store the attributes anyway

Is this change in the above fork, or somewhere squirrelled away in Aussie land?

[autoSteve]

Then I have done a good thing, @gcoan.

Exclusions apply for Solcast in recorder. I'd definitely exclude any breakdowns, too... We appear to get away with the existing attribute load as of now.

def exclude_attributes(hass: HomeAssistant) -> set[str]:
    """Exclude potentially large attributes from being recorded in the database."""
    return {"detailedForecast", "detailedHourly", "hard_limit"}

[autoSteve]

Is this change in the above fork branch, or somewhere squirrelled away in Aussie land?

Yeah, nah. Not exclusive to me. I sent it to the branch!

[gcoan]

Just downloaded the branch, replaced all the entire solcast-solar directory and restarted HA.

Messages in the log about upgrading to options v8:

2024-07-15 21:14:25.741 DEBUG (MainThread) [custom_components.solcast_solar] Options version 7
2024-07-15 21:14:25.743 DEBUG (MainThread) [custom_components.solcast_solar] Upgraded to options version 8
2024-07-15 21:14:25.743 INFO (MainThread) [custom_components.solcast_solar]
-------------------------------------------------------------------
Solcast integration version: 4.0.38
This is a custom integration. When troubleshooting a problem, after
reviewing open and closed issues, and the discussions, check the
required automation is functioning correctly and try enabling debug
logging to see more. Troubleshooting tips available at:
https://github.com/BJReplay/ha-solcast-solar/discussions/38
Beta versions may also have addressed some issues so look at those.
If all else fails, then open an issue and our community will try to
help: https://github.com/BJReplay/ha-solcast-solar/issues
-------------------------------------------------------------------
2024-07-15 21:14:25.759 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites cache exists
2024-07-15 21:14:25.760 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Connecting to - https://api.solcast.com.au/rooftop_sites?format=json&api_key=******Db75Av
2024-07-15 21:14:26.163 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 200/Success in sites_data()
2024-07-15 21:14:26.164 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing sites cache
2024-07-15 21:14:26.186 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites data: {'sites': [{'name': '87 High Street (E)', 'resource_id': 'bc31-4d4c-e844-6a4e', 'capacity': 10, 'capacity_dc': 8.58, 'longitude': **.******, 'latitude': **.******, 'azimuth': -100, 'tilt': 37, 'install_date': '2023-01-07T00:00:00.0000000Z', 'loss_factor': 0.99, 'tags': ['UK', 'Bedfordshire', 'GivEnergy']}, {'name': '87 High Street (W)', 'resource_id': '23bb-0216-85c6-fd94', 'capacity': 9, 'capacity_dc': 6.34, 'longitude': **.******, 'latitude': **.******, 'azimuth': 80, 'tilt': 37, 'install_date': '2023-01-07T00:00:00.0000000Z', 'loss_factor': 0.98, 'tags': ['UK', ' Bedfordshire', ' GivEnergy']}], 'page_count': 1, 'current_page': 1, 'total_records': 2}
2024-07-15 21:14:26.186 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Getting API limit and usage from solcast for ******Db75Av
2024-07-15 21:14:26.186 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] API usage cache exists
2024-07-15 21:14:26.520 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 200/Success in sites_usage()
2024-07-15 21:14:26.520 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing API usage cache file: /config/solcast-usage.json
2024-07-15 21:14:26.530 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] API counter for ******Db75Av is 8/10
2024-07-15 21:14:26.654 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] The saved data file exists: file type is <class 'dict'>
2024-07-15 21:14:27.057 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Calc forecast start index midnight utc: 2024-07-14 23:00:00+00:00, idx 14145, len 14530
2024-07-15 21:14:27.143 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-15 contains all 48 records
2024-07-15 21:14:27.144 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-16 contains all 48 records
2024-07-15 21:14:27.144 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-17 contains all 48 records
2024-07-15 21:14:27.145 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-18 contains all 48 records
2024-07-15 21:14:27.145 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-19 contains all 48 records
2024-07-15 21:14:27.147 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-20 contains all 48 records
2024-07-15 21:14:27.148 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-21 contains all 48 records
2024-07-15 21:14:27.148 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Data for 2024-07-22 contains all 48 records
2024-07-15 21:14:27.148 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Build forecast processing took 0.4937s
2024-07-15 21:14:27.149 INFO (MainThread) [custom_components.solcast_solar.solcastapi] Loaded solcast.json forecast cache
2024-07-15 21:14:27.155 DEBUG (MainThread) [custom_components.solcast_solar.coordinator] Finished fetching solcast_solar data in 0.000 seconds (success: True)

Then went and looked at the PV forecast today and was surprised to see the per site estimates already present - I guess since the data is already in the solcast.json file it didn't need an API call (and I'd saved one back) to download the data:

new integration config option, nice !

Is it worth having sub-options for enabling the different estimate breakdowns (standard, 10 and 90)? I'd only want the 10, the standard duplicates the main sensor value (although of course the per-site standard is useful)

Also, since we are adding configurability, how about the option to turn off the detailedForecast and detailedHourly? At least one person didn't have a use for these, I do as Predbat needs the detailedForecast but the detailedHourly, nah.

Have played around turning the options on and off and it all seems to work as expected. I checked the per day, peak forecast, power now and they all looked good in terms of configurability.

Ran a scheduled solcast poll update and all looks OK in the log. I did notice in the log that it seems every time I changed the config options it re-initialised the integration completely, so repeated sets of log messages like this:

2024-07-15 21:32:27.439 INFO (MainThread) [custom_components.solcast_solar]
-------------------------------------------------------------------
Solcast integration version: 4.0.38
This is a custom integration. When troubleshooting a problem, after
reviewing open and closed issues, and the discussions, check the
required automation is functioning correctly and try enabling debug
logging to see more. Troubleshooting tips available at:
https://github.com/BJReplay/ha-solcast-solar/discussions/38
Beta versions may also have addressed some issues so look at those.
If all else fails, then open an issue and our community will try to
help: https://github.com/BJReplay/ha-solcast-solar/issues
-------------------------------------------------------------------
2024-07-15 21:32:27.446 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites cache exists
2024-07-15 21:32:27.446 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Connecting to - https://api.solcast.com.au/rooftop_sites?format=json&api_key=******Db75Av
2024-07-15 21:32:28.066 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] HTTP session returned status 200/Success in sites_data()
2024-07-15 21:32:28.072 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Writing sites cache
2024-07-15 21:32:28.111 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] Sites data: {'sites': [{'name': '87 High Street (E)', 'resource_id': 'bc31-4d4c-e844-6a4e', 'capacity': 10, 'capacity_dc': 8.58, 'longitude': **.******, 'latitude': **.******, 'azimuth': -100, 'tilt': 37, 'install_date': '2023-01-07T00:00:00.0000000Z', 'loss_factor': 0.99,

blah blah, does a full sites download etc again

[gcoan]

Been able to create the template sensors I wanted for total energy today per site, thanks

[autoSteve]

What lovely feedback to receive in the morning!

It is so annoying that any options change reloads the integration from scratch. I can see why it does it, and it is by design, because I looked into whether it would be possible to stop the behaviour for some settings changes. It is what it is for now.

Great suggestions. I don't use the detailed attributes, and having them makes for an ungainly attributes view in the UI for the forecast today.

[autoSteve]

Something like this, @gcoan?

[gcoan]

Chef's kiss 😙

[autoSteve]

The attribute options are now more than a pretty face. They're functional. I'll test a bit more then commit to the branch.

[autoSteve]

Options will update again, Geoff, to version 9, which will mean a dual update for others 7->8->9.

I'm working out whether I can consolidate these into a single v8.

[gcoan]

Options will update again, Geoff, to version 9, which will mean a dual update for others 7->8->9.

I'm working out whether I can consolidate these into a single v8.

If I upgrade to this latest branch of options 9 and you rewrite history am I going to have problems then as I'll be v9 and the code expects v8?

Should I hold off until you decide what to do?

[autoSteve]

Mornin'.

Good point.

I'm trying to rig up an alpha release that will extend the v8 options, not advance the version counter. When it becomes a big grown up release I will remove that code that holds back the version. Essentially changing a temporary <= 8 to a < 8.