BJReplay / ha-solcast-solar

Solcast Integration for Home Assistant
Apache License 2.0
177 stars 34 forks source link

Cannot connect to Solcast, so the integration will obviously fail #80

Closed pagasam closed 3 months ago

pagasam commented 3 months ago

The problem

Hello everyone

Hopefully I'm asking my question in the right place. I have successfully downloaded and installed the Solcast integration. A corresponding API query automation is running in the background and the solcast.json is present. Unfortunately, I am not getting any values in the Solcast service entities. Reloading the Solcat integration several times has not been successful so far. Do you have any idea where I could start to fix this?

Screenshot 2024-07-09 125842

Thank you very much for your help

What version of Solcast Integration has the issue?

v4.0.35

What version of Home Assistant are you using?

No response

Diagnostics information

No response

Logs debug/info/error output

2024-07-09 11:57:38.070 INFO (MainThread) [custom_components.solcast_solar] SOLCAST - Service call: update_forecasts
2024-07-09 11:57:38.072 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - http_data error: Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 1013, in buildforecastdata
    self._forecasts_start_idx = self.calcForecastStartIndex()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/solcast_solar/solcastapi.py", line 1031, in calcForecastStartIndex
    _LOGGER.debug("SOLCAST - calcForecastStartIndex midnight_utc %s, idx %s, len %s", midnight_utc, idx, len(self._data_forecasts))
                                                                                                    ^^^
UnboundLocalError: cannot access local variable 'idx' where it is not associated with a value

2024-07-09 11:57:38.077 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.077 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 0 st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.078 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.078 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_time_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 1 st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_time_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.080 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-10 22:00:00 end 2024-07-11 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.080 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 2 st 2024-07-10 22:00:00 end 2024-07-11 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-11 22:00:00 end 2024-07-12 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 3 st 2024-07-11 22:00:00 end 2024-07-12 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-12 22:00:00 end 2024-07-13 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 4 st 2024-07-12 22:00:00 end 2024-07-13 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-13 22:00:00 end 2024-07-14 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 5 st 2024-07-13 22:00:00 end 2024-07-14 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-14 22:00:00 end 2024-07-15 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.084 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 6 st 2024-07-14 22:00:00 end 2024-07-15 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:58:00.158 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 11:59:00.157 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 12:00:00.156 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 12:00:00.193 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.194 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_remaining_today st 2024-07-09 10:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.194 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 11:00:00 end 2024-07-09 12:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_custom_hours st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 09:50:00 end 2024-07-09 10:10:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:20:00 end 2024-07-09 10:40:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:50:00 end 2024-07-09 11:10:00 st_i 0 end_i 0 res 0

2024-07-09 12:05:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_remaining_today st 2024-07-09 10:05:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 11:00:00 end 2024-07-09 12:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_custom_hours st 2024-07-09 10:05:00 end 2024-07-09 11:05:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 09:55:00 end 2024-07-09 10:15:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.197 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:25:00 end 2024-07-09 10:45:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.197 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:55:00 end 2024-07-09 11:15:00 st_i 0 end_i 0 res 0

Additional information

No response

autoSteve commented 3 months ago

What's in the solcast.json file? A debug log of the integration starting up would be handy.

That error message is pretty rough under the circumstances, and wants cleaning up, so sorry about that.

Also, what's in solcast-sites.json file? (Please redact your latitude/longitide.)

autoSteve commented 3 months ago

The error 'symptom' is because the forecasts have not been able to be built, so it follows that there is no data from Solcast.

The only way for this to happen and return that unfortunate error is if there are no Solcast sites to iterate. (I need to fix this, because no Solcast sites should then not attempt to build a forecast, and this should clearly be spelled out...)

So we need to know what is in solcast-sites.json (GPS location redacted if it's in there), so need a log of the initial start of the integration.

On startup, the sites are retrieved. This can sometimes be initially problematic because of the Solcast API responding with "429/Too busy" for hobbyists, hence the need to repetitively restart if a solcast-sites.json has not been created (which is a cache, to expect future "429/Too busy" failures).

pagasam commented 3 months ago

solcast.json: {"siteinfo": {}, "last_updated": "2024-07-09T11:32:17.010580+00:00", "version": 4}

solcast-sites.json file does not exist

home-assistant.log.txt

autoSteve commented 3 months ago

This has exposed an interesting issue, by the way.

Should there be no sites in the cache, and the sites fail to load, then on the next call to update the forecast (in the present version) the solcast.json will be written, as possibly an empty file containing {}. This is "possibly" empty because there is an initial call to load this file into memory, so writing it should cause no harm. But if the load call fails too for some reason?

I have already generated a fix for the attempt to build a forecast where no sites have been attempted, so the "idx" error shoud not recurr in 4.0.next.

autoSteve commented 3 months ago

There's the problem. A connection timeout. You just need to work out why your HA cannot access solcast.com, which if that were the case would result in a timeout. Solcast are usually pretty quick to connect, even if the conversation ends abruptly with a 429/Too busy, and that's not occurring in this case.

The integration must have the site detail to go get forecasts.

SOLCAST - sites_data get sites timed out, attempting to continue
SOLCAST - No cached site(s) are available for ******EBBmrt to cope with Solcast API timeout
SOLCAST - sites_data Timed out async wait for connection to solcast server, and one or more site caches failed to load. This is critical, and the integration cannot function reliably. Suggestion: Attempt integration reload.
SOLCAST - sites_usage Connection Error - Timed out connection to solcast server
SOLCAST - load_saved_data site count is zero. The get sites must have failed.
pagasam commented 3 months ago

How do I check or provide access from HA to solcast.com? How do I provide the location details to the integration?

autoSteve commented 3 months ago

How do I check or provide access from HA to solcast.com?

Can your HA server ping solcast.com? I don't think there are any TCP testing commands available like telnet, should this be a HAOS install. You don't specify.

How do I provide the location details to the integration?

You don't. You can't contact solcast.com, so it's a moot point.

The integration gets the sites using your API key.

pagasam commented 3 months ago

By the way -> thank you very much for the answers to my questions so far.

Screenshot 2024-07-09 143945

autoSteve commented 3 months ago

What do you get if you execute:

curl -X GET -H 'content-type: application/json' https://api.solcast.com.au/rooftop_sites?api_key=YOURAPIKEYHERE

You should get something like:

~ # curl -X GET -H 'content-type: application/json' https://api.solcast.com.au/rooftop_sites?api_key=bRFWtt3wxxxxxxxxxxxxxxusNV8ggJ
{"sites":[{"name":"Iris North-East","resource_id":"b68d-c05a-c2b3-2cf9","capacity":6.857,"capacity_dc":7.92...
pagasam commented 3 months ago

I get an answer back in the way described above.

autoSteve commented 3 months ago

Just to confirm, this curl was performed on the server that is running Home Assistant, yes?

All I know is that it is integration v4.0.35, and you have a networking issue that no one else has, and we do not have an integration issue that I can see.

Right now I have absolutely nothing to go on. The call works, so it should work from the integration, hence should write the solcast-sites.json, and then move on to a forecast get.

pagasam commented 3 months ago

Yes, this curl was executed on the server running Home Assistant.

Core 2024.7.1 Supervisor 2024.06.2 Operating System 12.4

runs on an intel NUC

autoSteve commented 3 months ago

Thanks for that. That's exactly the same as I'm running, except Proxmox VM for me.

I'm really stumped here, @pagasam, and can't think what could be causing a failure to connect to api.solcast.com.au, especially as a curl from HAOS works fine.

The logged timeout is genuine, 60 seconds in both cases, so it's not like the code is prematurely declaring a timeout, and I assume that the curl returned near instantaneously.

There is a beta upgrade available to v4.0.37, but I honestly don't think that's going to make any difference. That release focuses on some new attributes, plus improvements to exception handling. You could try a re-download/restart, but I can't see how it would help.

autoSteve commented 3 months ago

I wonder if this could possibly somehow be IPv6 related. If DNS returns a v6 address to use then your v6 config in your router could be weird. DNS may have returned v4 for the curl.

You could try turning v6 off in the HA network configuration.

(edit) dig -6 ... doesn't work for me, as I have no local v6 DNS configured, so HA would always use v4.

(edit) Are any other Internet-dependant integrations failing?

pagasam commented 3 months ago

You were right. It was the v6 configuration. I have now deactivated this and only use v4. The integration now runs smoothly with the available beta upgrade to v4.0.37.

Thank you very much for your help, I really appreciate it

autoSteve commented 3 months ago

Happy to be an Agony Aunt for non-related issues, @pagasam...

Do please enjoy. And especially enjoy v4.0.37. We're building an even greater integration, with more activity, and more thought, commit by commit.