Netatmo cameras are offline

[SMLMG]

My 5 Netatmo cameras are offline more than online

I would like my cameras to be in HA. I have always checked my Netatmo app on IOS to make sure everything is working and it always is. I do not know what is causing me to not see the cameras in HASS. ## Environment Home Assistant 0.110.3, Frontend version: 20200519.5 - latest arch | armv7l -- | -- chassis | embedded dev | false docker | true hassio | true host_os | HassOS 3.13 installation_type | Home Assistant os_name | Linux os_version | 4.19.114-v7 python_version | 3.7.7 supervisor | 225 timezone | America/Chicago version | 0.110.3 virtualenv | false - Home Assistant Core release with the issue: 103.3 - Last working Home Assistant Core release (if known): Unknown - has been going on for a month or more. - Operating environment (Home Assistant/Supervised/Docker/venv): Docker - Integration causing this issue: Netatmo - Link to integration documentation on our website: https://www.home-assistant.io/integrations/netatmo/ ## Problem-relevant `configuration.yaml`

netatmo:
  client_id: !secret netatmo_client_id_key
  client_secret: !secret netatmo_secret_key

Traceback/Error logs

2020-05-29 10:49:49 WARNING (MainThread) [homeassistant.components.climate] Setup of climate platform netatmo is taking over 10 seconds.
2020-05-29 10:49:54 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform netatmo is taking over 10 seconds.
2020-05-29 10:49:56 WARNING (MainThread) [homeassistant.components.camera] Setup of camera platform netatmo is taking over 10 seconds.
2020-05-29 10:50:39 ERROR (MainThread) [homeassistant.components.climate] Setup of platform netatmo is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2020-05-29 10:50:43 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 989, in create_connection
    ssl_handshake_timeout=ssl_handshake_timeout)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1017, in _create_connection_transport
    await waiter
ConnectionResetError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/netatmo/climate.py", line 136, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/climate.py", line 119, in get_entities
    home_data.setup()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/climate.py", line 428, in setup
    self.homedata = pyatmo.HomeData(self.auth)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/thermostat.py", line 26, in __init__
    resp = self.authData.post_request(url=_GETHOMESDATA_REQ)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 148, in post_request
    resp = query(url, params, timeout, 3)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 142, in query
    self._oauth.token = self.refresh_tokens()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/api.py", line 32, in refresh_tokens
    self.session.async_ensure_token_valid(), self.hass.loop
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 414, in async_ensure_token_valid
    new_token = await self.implementation.async_refresh_token(self.token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 79, in async_refresh_token
    new_token = await self._async_refresh_token(token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 152, in _async_refresh_token
    "refresh_token": token["refresh_token"],
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 166, in _token_request
    resp = await session.post(self.token_url, data=data)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.netatmo.com:443 ssl:None [None]
2020-05-29 10:50:44 ERROR (MainThread) [homeassistant.components.sensor] Setup of platform netatmo is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2020-05-29 10:50:45 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 989, in create_connection
    ssl_handshake_timeout=ssl_handshake_timeout)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1017, in _create_connection_transport
    await waiter
ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 146, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 133, in get_entities
    dc_data = data_class(auth)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/weather_station.py", line 24, in __init__
    resp = self.authData.post_request(url=self.urlReq)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 148, in post_request
    resp = query(url, params, timeout, 3)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 142, in query
    self._oauth.token = self.refresh_tokens()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/api.py", line 32, in refresh_tokens
    self.session.async_ensure_token_valid(), self.hass.loop
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 414, in async_ensure_token_valid
    new_token = await self.implementation.async_refresh_token(self.token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 79, in async_refresh_token
    new_token = await self._async_refresh_token(token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 152, in _async_refresh_token
    "refresh_token": token["refresh_token"],
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 166, in _token_request
    resp = await session.post(self.token_url, data=data)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.netatmo.com:443 ssl:None [None]
2020-05-29 10:50:46 ERROR (MainThread) [homeassistant.components.camera] Setup of platform netatmo is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2020-05-29 10:50:46 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 989, in create_connection
    ssl_handshake_timeout=ssl_handshake_timeout)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1017, in _create_connection_transport
    await waiter
ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 65, in async_setup_entry
    async_add_entities(await hass.async_add_executor_job(get_entities), True)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 53, in get_entities
    for camera in camera_data.get_all_cameras():
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 237, in get_all_cameras
    self.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 239, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 261, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 30, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 148, in post_request
    resp = query(url, params, timeout, 3)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 142, in query
    self._oauth.token = self.refresh_tokens()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/api.py", line 32, in refresh_tokens
    self.session.async_ensure_token_valid(), self.hass.loop
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 414, in async_ensure_token_valid
    new_token = await self.implementation.async_refresh_token(self.token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 79, in async_refresh_token
    new_token = await self._async_refresh_token(token)
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 152, in _async_refresh_token
    "refresh_token": token["refresh_token"],
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 166, in _token_request
    resp = await session.post(self.token_url, data=data)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.netatmo.com:443 ssl:None [None]

Additional information

Sorry to bother you with this ... I was hoping one of the releases would resolve issue. Thank you.

[probot-home-assistant[bot]]

Hey there @cgtobi, mind taking a look at this issue as its been labeled with a integration (netatmo) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

[SMLMG]

Message from Netatmo received today ...

Webhooks Status: Your app has been webhook banned. To avoid being banned, your application must always return '200' and answer quickly to our server (< 10sec). If your application did not answer quick enough or if it returned another code more than five times in a row, your webhook will be deactivated during 24 hours. To manually unban your app, click on the button below.

[cgtobi]

Can you please share some details about your setup? Are you behind a proxy?

[SMLMG]

Message from Netatmo again just now received today ...

Webhooks Status: Your app has been webhook banned. To avoid being banned, your application must always return '200' and answer quickly to our server (< 10sec). If your application did not answer quick enough or if it returned another code more than five times in a row, your webhook will be deactivated during 24 hours. To manually unban your app, click on the button below.

[SMLMG]

Can you please share some details about your setup? Are you behind a proxy?

I replied in the community ... Was that of any help or do you need more information? I am happy to help you in any way I can! I think the only thing I left out was the camera configuration - there are 4 outdoor cameras forming a perimeter around my house and one indoor (Welcome) camera for intruders that set off the alarm. I also wrote to Netatmo to see if they could give me a report for all the server disconnects on our end.

[SMLMG]

The integration is behaving better. Less disconnects. I do not know if you have done anything with this but the performance seems better.

[SMLMG]

Here is Netatmo’s reply to my request for statistics on bans -

From: "Leslie (Netatmo)" contact-api@netatmo.com Date: June 2, 2020 at 3:58:38 AM CDT To: Stephen Mark smlmark@lucraconsulting.com Subject: NETATMO // Ticket 2657880 Reply-To: Netatmo contact-api@netatmo.com

Hello Stephen,

Your webhook is banned if it doesn't return a 200 OK code 5 times in a row and under 10 seconds. You can manually ask for a deban 10 times in 24 hours Unfortunately, we don't have access to logs concerning webhooks for GDPR compliance reasons I advise you to verify if the server of the reply url of your webhook is correctly configured

Have a good day, Leslie - Community Manager

What is the server of my reply for HASS? It can’t be me because my app cannot be deployed if not configured correctly. Anyway, at least you have a contact at Netatmo should you need one.

[SMLMG]

Cameras went south again just now. Anything I can do to help?

Webhooks

Status: Your app has been webhook banned. To avoid being banned, your application must always return '200' and answer quickly to our server (< 10sec). If your application did not answer quick enough or if it returned another code more than five times in a row, your webhook will be deactivated during 24 hours. To manually unban your app, click on the button below.

[SMLMG]

Cameras went south three times in less than 12 hours ... Any ideas

Webhooks

Status: Your app has been webhook banned. To avoid being banned, your application must always return '200' and answer quickly to our server (< 10sec). If your application did not answer quick enough or if it returned another code more than five times in a row, your webhook will be deactivated during 24 hours. To manually unban your app, click on the button below.

[cgtobi]

Well, since this is not only affecting Netatmo I assume something with your setup is not correctly configured. I have two instances, one with Nabu Casa and one behind a reverse proxy. So please share you proxy and http config. If you like you can contact me via discord to discuss this more privately.

[cgtobi]

Hi, this might be difficult as we're on completely different time zones apparently. Just ping me on Discord and let's find out.

I deleted your last message as it contained sensitive information.

[cgtobi]

Please check the full Nginx Reverse Proxy guide here.

I think you're config is missing this:

http:
  use_x_forwarded_for: true
  trusted_proxies: <NGINX IP address here, or 127.0.0.1 if hosted on the same machine>

You probably have to edit your proxy settings as well.

Not sure if this is the only issue.

[SMLMG]

We are in different time zones. However, I cannot sleep so if you are around let’s discuss. One thing I should have mentioned is that I have a static IP coming into the house from my ISP. The ISP requires this setup. I do not see how this would change anything I did. TTS is working fine again. It needed an internal url once I implemented the Nginx add-on.

[SMLMG]

BTW - you understand that I am using the Nginx add-on right? From what I saw the only thing I needed to do was enter the external url and port forward 443 to 443 for my PI’s IP address. I know nothing of Nginx but was convinced to use this addon to have another avenue to access the PI after commenting that having an internal url seemed idiotic once you set up remote access. That is, you could only access the Pi from an external address anyway. There is a community moderator, Petro, who guided my efforts to set up the Nginx add-on. My statement was ignorant because TTS did not function correctly without a correct internal url. Finally, I only recently implemented the Nginx addon. The pattern of losing Netatmo due to Server request timeouts was happening before the addon was implemented. I do not believe the server timing out is mine. My conversation with Netatmo’s engineer indicated that whatever program and hardware was directly issuing the webhook was the problem. Is that my Pi?

[SMLMG]

The guide you posted is for home-assistant core users. The ones that still want to control the machine in other words. I am a HASS OS or HASS IO user so the document does not apply to me. All the Nginx voodoo is handled by the addon. Where do we go from here?

[cgtobi]

I don't know the Nginx add-on. I just know that you need a reverse proxy if you don't want to expose your whole instance to the web. And the settings for that are in the article linked above.

[cgtobi]

Which add-on are you using?

To my understanding it is required anyway to add the config to http: as stated above.

http:
  use_x_forwarded_for: true
  trusted_proxies: <NGINX IP address here, or 127.0.0.1 if hosted on the same machine>

[SMLMG]

That is not my understanding. The addon is called NGINX Home Assistant SSL proxy. It handles everything beautifully and is pure magic. How about this - would it work if I build a separate HASS instance with just cameras and no proxy? Then if they go offline will you take a longer look? You seem awful skeptical that anything is wrong but as soon as I posted this another user jumped right on board. I don't mind being the low man on the totem pole that points out bugs but it is a very useful service.

[cgtobi]

Don't get me wrong, I am happy for any bug that is found and can be fixed. The issue is just that it is pretty difficult to replicate and to debug this way.

[SMLMG]

I assume you do not have a Netatmo camera right? Also, probably not many people have five like me. Maybe the infrastructure on the HASS side cannot keep up with the requests to my little Pi? I'm really guessing since I do not know the program internals or the HW infrastructure. What I can tell you is that on the old Home Asssistant it ran perfectly. Should I put out a request to the community for all Netatmo users experiencing webhook bans to step forward to get a sense if there's any scope to the problem?

[SMLMG]

BTW - it can go a day sometimes and be fine.

[SMLMG]

If you do not even have a Netatmo camera to play with then it will be hard to do unless it is a glaring error in the code base. However, it used to work flawlessly just months ago. However, that seems like an eternity now. I am happy to build another instance with just Netatmo cameras and get rid of Nginx addon as we seem to disagree a bit on it. You let me know what you want since you have no "skunkworks" to debug these things. You would think they would assign someone with the equipment but what do I know. Not a good position for either of us.

[cgtobi]

I do have one indoor camera myself and nobody has assigned me to this. This is open source and I am the code owner of the Netatmo integration as well as a core contributor to the underlying module. So I am doing this all in my spare time.

Please jump on to discord to discuss how we get me access to your setup securely and privately.

[SMLMG]

I’m so sorry that the word assigned offended you. However, your reaction to it, told me everything I need to know about you. This is not going to work out. You feel you are doing me some kind of favor by working on this issue. You are not doing anything as it is still broken and I am still writing to you. I am not going to chase you down on Discord. You can fix it or not as you see fit. As the code owner you should assume full responsibility for the error and, out of pride alone, make an honest effort to fix it. If you do not have the “spare” time to give it the proper attention it needs then you should relinquish your role. If that does not suit you then get someone senior to give assistance. I’ve done exactly what is asked of people that report bugs. Others have commented on this same problem. I’m done. I’d wish you good luck but you’ll need to lose your thin skin to find any success in this life.

[pvizeli]

Please read https://github.com/home-assistant/architecture/blob/master/adr/0008-code-owners.md first. However -> https://www.home-assistant.io/code_of_conduct/

You get a cooldown of 30 days to remember what our CoC said