New Netatmo integration looses entities after HA restart

ma0git commented 4 years ago

The problem

Since the recent refactoring of the Netatmo integration in 0.105 the Netatmo entities often show up as "Unavailable" after a restart of HA. To fix the problem I need to remove the Netatmo integration and add it again. The problem does not appear after every restart of HA. Usually if I restart HA several times shortly after I fixed the problem, the problem does not appear again. However, when I restart HA after it was running for a few hours the entities always disappear. If the problem once appeared it wont go away by a restart of HA, so really only removing and adding the Netatmo integration fixes it.

I am not sure, if this problem is related to the fact that I have to rename all the entities in the HA registry manually. I have to do this because the default entity names seem to have changed with the refactored Netatmo integration. Because I am recording the values in an InfluxDB I need the entity names to stay the same as before because it seems also not possible to rename the recorded names in InfluxDB easily.

Environment

Home Assistant release with the issue: 0.106.1
Last working Home Assistant release (if known): 0.104
Operating environment (Hass.io/Docker/Windows/etc.): RPi with manually installed HA
Integration causing this issue: netatmo
Link to integration documentation on our website: https://www.home-assistant.io/integrations/netatmo/

Problem-relevant `configuration.yaml`

netatmo:
  client_id: hidden
  client_secret: hidden

Traceback/Error logs

In the log I see the following after the problem appears:

2020-02-29 11:19:10 ERROR (MainThread) [homeassistant.components.binary_sensor] Error while setting up netatmo platform for binary_sensor
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/binary_sensor.py", line 95, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/binary_sensor.py", line 60, in get_entities
    for camera in data.get_all_cameras():
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 258, in get_all_cameras
    self.update()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 282, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/camera.py", line 28, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomedata'
2020-02-29 11:19:10 ERROR (MainThread) [homeassistant.components.camera] Error while setting up netatmo platform for camera
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 64, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 52, in get_entities
    for camera in camera_data.get_all_cameras():
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 258, in get_all_cameras
    self.update()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 282, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/camera.py", line 28, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomedata'
2020-02-29 11:19:11 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up netatmo platform for sensor
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/sensor.py", line 128, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/sensor.py", line 115, in get_entities
    dc_data = data_class(auth)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/home_coach.py", line 15, in __init__
    super(HomeCoachData, self).__init__(authData, urlReq=_GETHOMECOACHDATA_REQ)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/weather_station.py", line 24, in __init__
    resp = self.authData.post_request(url=self.urlReq)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomecoachsdata'

Additional information

probot-home-assistant[bot] commented 4 years ago

Hey there @cgtobi, mind taking a look at this issue as its been labeled with a integration (netatmo) you are listed as a codeowner for? Thanks!

cgtobi commented 4 years ago

Could you please switch logging for the netatmo integration to debug to gather more information?

ma0git commented 4 years ago

The log I get with debug level enabled when it fails is this here:

2020-02-29 18:52:10 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.climate] No thermostat devices available
2020-02-29 18:52:10 ERROR (MainThread) [homeassistant.components.binary_sensor] Error while setting up netatmo platform for binary_sensor
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/binary_sensor.py", line 95, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/binary_sensor.py", line 60, in get_entities
    for camera in data.get_all_cameras():
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 258, in get_all_cameras
    self.update()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 282, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/camera.py", line 28, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomedata'
2020-02-29 18:52:10 ERROR (MainThread) [homeassistant.components.camera] Error while setting up netatmo platform for camera
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 64, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 52, in get_entities
    for camera in camera_data.get_all_cameras():
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 258, in get_all_cameras
    self.update()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/camera.py", line 282, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/camera.py", line 28, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomedata'
2020-02-29 18:52:14 DEBUG (SyncWorker_15) [homeassistant.components.netatmo.sensor] weather station detected!
2020-02-29 18:52:14 DEBUG (SyncWorker_15) [homeassistant.components.netatmo.sensor] Adding module Büro 70:ee:50:05:73:58
2020-02-29 18:52:14 DEBUG (SyncWorker_15) [homeassistant.components.netatmo.sensor] Adding module Aussen 02:00:00:05:6e:d6
2020-02-29 18:52:14 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up netatmo platform for sensor
Traceback (most recent call last):
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/sensor.py", line 128, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/homeassistant/lib/python3.7/site-packages/homeassistant/components/netatmo/sensor.py", line 115, in get_entities
    dc_data = data_class(auth)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/home_coach.py", line 15, in __init__
    super(HomeCoachData, self).__init__(authData, urlReq=_GETHOMECOACHDATA_REQ)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/weather_station.py", line 24, in __init__
    resp = self.authData.post_request(url=self.urlReq)
  File "/opt/homeassistant/lib/python3.7/site-packages/pyatmo/auth.py", line 122, in post_request
    raise ApiError(f"Error when accessing '{url}'")
pyatmo.exceptions.ApiError: Error when accessing 'https://api.netatmo.com/api/gethomecoachsdata'

After I fixed it again by removing and adding the integration it looks like this:

2020-02-29 19:01:08 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.climate] No thermostat devices available
2020-02-29 19:01:08 DEBUG (SyncWorker_0) [homeassistant.components.netatmo.binary_sensor] No camera entities to add
2020-02-29 19:01:09 DEBUG (SyncWorker_7) [homeassistant.components.netatmo.camera] No cameras found
2020-02-29 19:01:12 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.sensor] weather station detected!
2020-02-29 19:01:12 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.sensor] Adding module Büro 70:ee:50:05:73:58
2020-02-29 19:01:12 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.sensor] Adding module Aussen 02:00:00:05:6e:d6
2020-02-29 19:01:13 DEBUG (SyncWorker_1) [homeassistant.components.netatmo.sensor] No home coach entities found

cgtobi commented 4 years ago

Could you please run this selftest script to gather further information? Feel free to contact me on Discord to discuss this further for privacy reasons as the log contains more details about your home.

https://gist.github.com/cgtobi/5601aa88efba2cce30a98aeae50582a2

ma0git commented 4 years ago

I got the following output from the script:

Energy Public Camera WeatherStation HomeCoach Smokedetector

Checking for available weather stations...
  Büro (70:ee:50:05:73:58)
    Büro Büro (70:ee:50:05:73:58)
      Temperature - 22.4
      CO2 - 2051
      Humidity - 47
      Noise - 59
      Pressure - 1000.1
      wifi_status - 72
      min_temp - 19.9
      max_temp - 22.4
      reachable - True
    Büro Aussen (02:00:00:05:6e:d6)
      Temperature - 9.8
      Humidity - 74
      rf_status - 89
      battery_vp - 5788
      battery_percent - 91
      min_temp - 6.3
      max_temp - 14.1
      reachable - True
WeatherStation [OK]

Checking for available home coach stations...
warning, no home coach station available for testing

Checking for available cameras...
warning, no camera available for testing

Checking for available smoke detectors...
warning, no smoke detectors available for testing

Checking for available thermostats/valves...
warning, no energy devices available for testing

Checking for public data...
Public [OK]

Let me know if you need anything else.

cgtobi commented 4 years ago

~~The good thing, it doesn’t look like a HA problem. The question still is why is only your weather station reported.~~ Let’s discuss this on discord if possible.

ma0git commented 4 years ago

It looks like this was actually a problem of the Netatmo API and not HA. I still don't really understand what exactly caused the problem, but after I did the following the problem disappeared:

While I had the problem, I had two devices which used the same Netatmo App/client_id to query the Netatmo API. Now I created a second App/client_id for the second device.
I pressed the 'Reset keys' button on the Netatmo dev site for the first App/client_id which created a new client_id/client_secret for it.

Maybe this at least helps other people who see the same strange behavior.

home-assistant / core