Netatmo - Error during webhook registration

edenis875 commented 5 days ago

The problem

Hello,

My Bubendorff shutters are not visible from HA.

I suspect the following error during HA startup to be the root cause which I'm not able to solve: Error during webhook registration - ("400 - Bad request - Invalid url parameter (21) when accessing 'https://api.netatmo.com/api/addwebhook'",) Also, integration diagnostic file shows "webhook_registered false".

Own app has been created on Netatmo developer portal and Netatmo weather station devices are successfully imported which seemingly means that my token is valid. As described in the documentation, I ensured 'redirect URI' and 'webhook URI' parameters were not set.

SSL is enabled on HA side; SSL port is opened on firewall side and forwarded to HA

I tried to

(temporarily) put my HA in DMZ, to ensure that there was no issue with my firewall
delete / re-import Netatmo integration
change external / internal HA URL ... but got no luck to remove above error message and have my shutters imported

Any idea of what is wrong here? Did I misconfigure anything? Is there any other useful information I could provide to help understanding where the issue comes from?

Many thanks,

What version of Home Assistant Core has the issue?

2024.9.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Netatmo

Link to integration documentation on our website

https://www.home-assistant.io/integrations/netatmo

Diagnostics information

config_entry-netatmo-01J7K24YR110VWKJHD5FC417N4.json config_entry-netatmo-01J7K24YR110VWKJHD5FC417N4.json config_entry-netatmo-01J7K24YR110VWKJHD5FC417N4.json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-09-12 14:19:36.102 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration tapo which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-09-12 14:19:36.108 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration tapo_control which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-09-12 14:19:36.110 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-09-12 14:19:53.969 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to load_verify_locations with args (<ssl.SSLContext object at 0xffff79406450>, '/usr/local/lib/python3.12/site-packages/pyoverkiz/overkiz-root-ca-2048.crt', None, None) inside the event loop by integration 'overkiz' at homeassistant/components/overkiz/__init__.py, line 239: return OverkizClient( (offender: /usr/local/lib/python3.12/ssl.py, line 708: context.load_verify_locations(cafile, capath, cadata)), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+overkiz%22
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#load_verify_locations
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 189, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component
    result = await _async_setup_component(hass, domain, config)
  File "/usr/src/homeassistant/homeassistant/setup.py", line 461, in _async_setup_component
    await asyncio.gather(
  File "/usr/src/homeassistant/homeassistant/setup.py", line 463, in <genexpr>
    create_eager_task(
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 45, in create_eager_task
    return Task(coro, loop=loop, name=name, eager_start=True)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 752, in async_setup_locked
    await self.async_setup(hass, integration=integration)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/usr/src/homeassistant/homeassistant/components/overkiz/__init__.py", line 64, in async_setup_entry
    client = create_local_client(
  File "/usr/src/homeassistant/homeassistant/components/overkiz/__init__.py", line 239, in create_local_client
    return OverkizClient(

2024-09-12 14:20:09.802 ERROR (MainThread) [homeassistant.components.netatmo] Error during webhook registration - ("400 - Bad request - Invalid url parameter (21) when accessing 'https://api.netatmo.com/api/addwebhook'",)
2024-09-12 15:00:10.551 ERROR (MainThread) [homeassistant.components.mobile_app.notify] Timeout sending notification to https://mobile-apps.home-assistant.io/api/sendPush/android/v1
2024-09-12 15:36:37.392 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 362, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 563, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message:
  Pause on PRI/Upgrade:

    b''
      ^
2024-09-12 15:57:20.348 DEBUG (MainThread) [pyatmo.auth] The Netatmo API returned b'{"error":{"code":7,"message":"Nothing to drop"}}' (406)
2024-09-12 15:57:20.349 DEBUG (MainThread) [homeassistant.components.netatmo] No webhook to be dropped
2024-09-12 15:57:20.349 INFO (MainThread) [homeassistant.components.netatmo] Unregister Netatmo webhook
2024-09-12 15:57:20.358 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher weather removed
2024-09-12 15:57:20.904 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher account added
2024-09-12 15:57:21.060 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher weather added
2024-09-12 15:57:21.161 DEBUG (MainThread) [pyatmo.helpers] Server response (tag: devices): {'body': {'devices': [], 'user': {'mail': 'exxxxxxxx@dxxxxxxxx.com', 'administrative': {'lang': 'fr', 'reg_locale': 'fr-FR', 'country': 'FR', 'unit': 0, 'windunit': 0, 'pressureunit': 0, 'feel_like_algo': 0}}}, 'status': 'ok', 'time_exec': 0.0185549259185791, 'time_server': 1726149441}
2024-09-12 15:57:21.162 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] No device data available
2024-09-12 15:57:21.162 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher air_care added
2024-09-12 15:57:21.323 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher home-645b86f3ff51c00xxxxxxxx added
2024-09-12 15:57:21.382 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher air_care removed
2024-09-12 15:57:21.575 DEBUG (MainThread) [pyatmo.auth] addwebhook: <ClientResponse(https://api.netatmo.com/api/addwebhook) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Thu, 12 Sep 2024 13:57:21 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Cache-Control': 'no-cache, must-revalidate', 'Expires': '0', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Powered-By': 'Netatmo', 'Content-Encoding': 'gzip')>

2024-09-12 15:57:21.579 INFO (MainThread) [homeassistant.components.netatmo] Register Netatmo webhook: https://exxxxxxxx.duckdns.org/api/webhook/29aa3463a54939cc8be4e914dedbc501xxxxxxxxxxxxxxxx
2024-09-12 15:58:20.947 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 299
2024-09-12 15:58:22.466 DEBUG (MainThread) [pyatmo.auth] dropwebhook: <ClientResponse(https://api.netatmo.com/api/dropwebhook) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Thu, 12 Sep 2024 13:58:22 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Cache-Control': 'no-cache, must-revalidate', 'Expires': '0', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Powered-By': 'Netatmo', 'Content-Encoding': 'gzip')>

2024-09-12 15:58:22.466 INFO (MainThread) [homeassistant.components.netatmo] Unregister Netatmo webhook
2024-09-12 15:58:22.478 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher weather removed
2024-09-12 15:58:22.597 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher account added
2024-09-12 15:58:22.740 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher weather added
2024-09-12 15:58:22.852 DEBUG (MainThread) [pyatmo.helpers] Server response (tag: devices): {'body': {'devices': [], 'user': {'mail': 'exxxxxxxx@dxxxxxxxx.com', 'administrative': {'lang': 'fr', 'reg_locale': 'fr-FR', 'country': 'FR', 'unit': 0, 'windunit': 0, 'pressureunit': 0, 'feel_like_algo': 0}}}, 'status': 'ok', 'time_exec': 0.030803918838500977, 'time_server': 1726149502}
2024-09-12 15:58:22.852 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] No device data available
2024-09-12 15:58:22.853 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher air_care added
2024-09-12 15:58:23.036 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher home-645b86f3ff51c003xxxxxxxx added
2024-09-12 15:58:23.095 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Publisher air_care removed
2024-09-12 15:58:27.188 DEBUG (MainThread) [pyatmo.auth] The Netatmo API returned b'{"error":{"code":21,"message":"Invalid url parameter"}}' (400)
2024-09-12 15:58:27.189 ERROR (MainThread) [homeassistant.components.netatmo] Error during webhook registration - ("400 - Bad request - Invalid url parameter (21) when accessing 'https://api.netatmo.com/api/addwebhook'",)
2024-09-12 15:59:22.719 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 298

Additional information

No response

home-assistant[bot] commented 5 days ago

Hey there @cgtobi, mind taking a look at this issue as it has been labeled with an integration (netatmo) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `netatmo` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign netatmo` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

netatmo documentation netatmo source _{^{(message by IssueLinks)}}

jetmcquack commented 4 days ago

Same here, thanks for reporting

xbeaudouin commented 3 days ago

Same here as well.

oywino commented 2 days ago

With so many people now reporting more or less exactly the same problem (here and in different other forums), we are eagerly awaiting feedback from @cgtobi 😶

Haifi58 commented 1 day ago

Netatmo integration worked for long time - now I cant acccess my devices:

Logger: homeassistant.components.netatmo Quelle: components/netatmo/init.py:151 Integration: Netatmo (Dokumentation, Probleme) Erstmals aufgetreten: 13:20:05 (1 Vorkommnisse) Zuletzt protokolliert: 13:20:05

Webhook not registered - https and port 443 is required to register the webhook

jetmcquack commented 1 day ago

Hello this is is a different issue. If you can’t reach port 443 from outside it’s possible that your reverse proxy has some issues. The issue in this thread is about the error “invalid parameters”

On 16 Sep 2024, at 13:23, Haifi58 @.***> wrote: Netatmo integration worked for long time - now I cant acccess my devices: Logger: homeassistant.components.netatmo Quelle: components/netatmo/init.py:151 Integration: Netatmo (Dokumentation, Probleme) Erstmals aufgetreten: 13:20:05 (1 Vorkommnisse) Zuletzt protokolliert: 13:20:05 Webhook not registered - https and port 443 is required to register the webhook

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

Haifi58 commented 1 day ago

Thanks for feedback

opened dev Account in netatmo
added credentials in home assistant
added netatmo Integration
could link Integration to dev Account
works now 😀

edenis875 commented 1 day ago

Hey there, I have been able to move ahead thanks to this thread: the webhook now looks successfully registered but I get plenty of weird errors messages afterwards (see log below) and my Bubendorff shutters are still not showing up. I'll keep digging but it would be much appreciated if anyone could help me fixing this... Many thanks,

2024-09-16 19:28:20.454 INFO (MainThread) [homeassistant.components.netatmo.data_handler] Netatmo webhook successfully registered
2024-09-16 19:29:14.716 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 282
2024-09-16 19:29:27.342 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 362, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 563, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message:
  Pause on PRI/Upgrade:

    b''
      ^
2024-09-16 19:30:11.939 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 208
2024-09-16 19:31:11.256 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 159
2024-09-16 19:31:17.139 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 362, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 563, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message:
  Pause on PRI/Upgrade:

    b''
      ^
2024-09-16 19:32:12.598 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 148
2024-09-16 19:33:11.396 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 131
2024-09-16 19:34:14.529 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 128
2024-09-16 19:35:13.137 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 119
2024-09-16 19:36:11.293 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 119
2024-09-16 19:37:14.924 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 112
2024-09-16 19:38:12.588 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 113
2024-09-16 19:43:10.994 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 87
2024-09-16 19:45:10.996 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 81
2024-09-16 19:46:11.139 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 79
2024-09-16 19:47:11.187 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 78
2024-09-16 19:48:12.481 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 80
2024-09-16 19:49:12.694 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 79
2024-09-16 19:50:12.597 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 81
2024-09-16 19:51:14.458 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 80
2024-09-16 19:52:12.131 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 82
2024-09-16 19:53:16.213 DEBUG (MainThread) [homeassistant.components.netatmo.data_handler] Calls per hour: 81

edenis875 commented 3 hours ago

One step further!

2024-09-17 19:44:48.104 INFO (MainThread) [homeassistant.components.netatmo.data_handler] Netatmo webhook successfully registered

No more errors BUT... my Bubendorff shutters remain not visible from Netatmo integration while they are from iDiamant! Also nothing is received when listening to netatmo_event from development tools section.

I guess something is still misconfigured, but I must say I have no further idea of what might be wrong, now... so if any one has any suggestion, it will be much appreciated!

Side note: I'll write details about settings changed somewhere here as soon as possible

oywino commented 1 hour ago

Here's my experience: I have two Home Assistant instances - one is HA-OS based using NabuCasa, the other is HA Container Core exposed to the internet through a Reverse Proxy. Both have the Netatmo integration and both have indoor as well as outdoor cameras. Netatmo Webhooks worked well with both for a long time. Then, sometime this summer, it started to get hickups. Now, neither of my HA instances are able to trigger on motion or any other event, except if a camera incidentally drops off-line, which some of them do from time to time (it's always been a bit unstable), not knowing why. They usually come back on-line again by themselves after a short while (less than an hour). However, those off-line/on-line events do generate a proper webhook trigger, not randomly, but persistently, so it seems the communication does work as it should. Why no other events comes through, is beyond me. On both my HA instances I can run the register/un-register Netatmo webhook service successfully. No error messages.

home-assistant / core