Forecast with force_hourly regularly unavaible — network error?

[diplix]

since version 2.0.3 i see regular errors like this for my force_hourly dwd weather instance (6 hour forcast continues to work flawlessly — no unavailable sensors at any time).

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.dwd_weather
Source: helpers/update_coordinator.py:290
Integration: Deutscher Wetterdienst (documentation, issues)
First occurred: 6. Oktober 2023 um 20:50:37 (9 occurrences)
Last logged: 05:31:20

Unexpected error fetching DWD Weather Coordinator for X104 data: '2023-10-16T23:00:00.000Z'
Unexpected error fetching DWD Weather Coordinator for X104 data: '2023-10-17T00:00:00.000Z'
Unexpected error fetching DWD Weather Coordinator for X104 data: '2023-10-17T01:00:00.000Z'
Unexpected error fetching DWD Weather Coordinator for X104 data: '2023-10-17T02:00:00.000Z'
Unexpected error fetching DWD Weather Coordinator for X104 data: '2023-10-17T03:00:00.000Z'
Traceback (most recent call last):
  File "/home/homeassistant/homeassistant_p3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/homeassistant_p3.11/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 246, in _async_update_data
    return await self.update_method()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/dwd_weather/connector.py", line 56, in async_update
    return await self._hass.async_add_executor_job(self._update)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/dwd_weather/connector.py", line 63, in _update
    self.dwd_weather.update(
  File "/home/homeassistant/homeassistant_p3.11/lib/python3.11/site-packages/simple_dwd_weatherforecast/dwdforecast.py", line 579, in update
    self.forecast_data[item]["wwP"] = temp_forecast_data[item]["wwP"]
                                      ~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: '2023-10-16T18:00:00.000Z'

sensors are „unavailable“ and recover aftre 5 minutes (next update). you can see those 5 minute „outages“ in history as well:

i have a feeling it’s rather some kind of timeout / timing issue, rather than a network problem since the pattern is too regular.

[FL550]

Yes, it seems so. The reason for crash is a change I made to make the precipitation duration and precipitation probability available in hourly mode. Normally this error shouldn't occur, but as two network requests are done for this there might be a timing issue. It seems like I need to handle this more gracefully. Thanks for reaching out!

[FL550]

Version v2.0.4 should fix this.

[nikipore]

Still same behavior in 2.0.4.

[diplix]

Still same behavior in 2.0.4.

can confirm.

[FL550]

Hm, same log files also?

[diplix]

i reverted to 2.0.2 when i became aware of the issue and did not look at the logs of 2.0.4. but i assumed the same error. looking at the history i see 10 minute gaps instead of 5 minute gaps, so the timing seems to have changed.

however, thinking about it, why not keep it all clean and mark sensors that are not available with hourly forecasts or certain stations as such? there is a clean and properly functioning workaround by configuring both a service with „force hourly“ and one without it and combining them as needed. i’m also thinking about stations that lack certain data (like Irradiance in #86).

missing sensor are marked as „unknown“ until version 2.0.2, why not mark them as „unavailable“ (once those sensors are activated), or even better, not show data/sensors at all that are not offered by certain dwd stations or forecast methods?

i’m happy to help document this in the readme.

[FL550]

however, thinking about it, why not keep it all clean and mark sensors that are not available with hourly forecasts or certain stations as such? there is a clean and properly functioning workaround by configuring both a service with „force hourly“ and one without it and combining them as needed.

I have implemented this.

i’m also thinking about stations that lack certain data (like Irradiance in #86). missing sensor are marked as „unknown“ until version 2.0.2, why not mark them as „unavailable“ (once those sensors are activated), or even better, not show data/sensors at all that are not offered by certain dwd stations or forecast methods?

This is a bit of a problem, because before data fetching I don't know which data is available. And the setup of the sensors happens async which means, I couln't check if the data point is available during setup without updating the whole data for the setup of each new sensor.

i’m happy to help document this in the readme.

Help is always welcome :)

[FL550]

This error might be due to missing data points in the future. I can try to avoid this by checking how much date keys are available. I'll fix this tomorrow.

[FL550]

Hopefully the new version removes this error.

[diplix]

seems to work for me now, no unavailable sensors anymore. thanks! i hope to find some time soon to make up to my promise to make some notes to the readme.

[FL550]

Glad to hear this! And thanks for the PR!