search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
74.01k stars 31.05k forks source link

Honeywell Integration API Rate Limited #88716

Closed mwjones1971 closed 1 year ago

mwjones1971 commented 1 year ago

The problem

Every few days, the integration will stop working, and I will get a request to reauthenticate. If I click the "reconfigure" button in Settings, I am presented with a generic dialog box and am unsure what to enter:

image

I have (as suggested in another thread) enabled debug level logging for homeassistant.components.honeywell and somecomfort, and have attached the results (redacting my login info). The key message I'm seeing is that the API is rate limited. If I reload the integration or restart HA, the problem seems to clear for the moment, but will return in a few days.

What version of Home Assistant Core has the issue?

2023.2.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Honeywell Total Connect Comfort (US)

Link to integration documentation on our website

https://www.home-assistant.io/integrations/honeywell

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Honeywell:
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
2023-02-21 13:56:43.389 WARNING (MainThread) [homeassistant.config_entries] Config entry 'honeywell' for honeywell integration not ready yet: Failed to initialize the Honeywell client: Connection error: maybe you have exceeded the API rate limit?; Retrying in background
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
2023-02-22 23:18:56.662 WARNING (MainThread) [homeassistant.components.climate] Updating honeywell climate took longer than the scheduled update interval 0:00:30
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 410, in async_update
2023-02-22 23:20:26.918 ERROR (MainThread) [somecomfort] Login as [user@domain} failed
2023-02-22 23:20:27.006 ERROR (MainThread) [somecomfort] Login as [user@domain] failed
2023-02-22 23:20:27.007 WARNING (MainThread) [homeassistant.config_entries] Config entry 'honeywell' for honeywell integration could not authenticate: Incorrect Password

Somecomfort:
2023-02-21 06:21:07.286 ERROR (MainThread) [somecomfort] API returned 500 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-21 09:18:38.172 ERROR (MainThread) [somecomfort] API returned 403 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-21 11:45:07.849 ERROR (MainThread) [somecomfort] API returned 200 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-21 13:56:39.147 ERROR (MainThread) [somecomfort] API Rate Limited at login.
2023-02-21 13:56:42.768 ERROR (MainThread) [somecomfort] Connection error 403
2023-02-21 13:56:43.384 ERROR (MainThread) [somecomfort] Connection error 500
2023-02-22 07:44:58.222 ERROR (MainThread) [somecomfort] API returned 200 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-22 08:56:55.617 ERROR (MainThread) [somecomfort] API returned 500 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 102, in _request_json
2023-02-22 23:19:26.924 ERROR (MainThread) [somecomfort] API returned 500 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-22 23:19:57.935 ERROR (MainThread) [somecomfort] API returned 200 from /portal/Device/CheckDataSession/7619001 request
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 152, in _get_thermostat_data
File "/usr/local/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 128, in _request_json
raise SomeComfortError("API returned %s, %s" % resp.status, req)
2023-02-22 23:20:26.838 ERROR (MainThread) [somecomfort] API Rate Limited at login.
2023-02-22 23:20:26.918 ERROR (MainThread) [somecomfort] Login as [user@domain] failed
2023-02-22 23:20:27.006 ERROR (MainThread) [somecomfort] Login as [user@domain] failed

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @rdfurman, @mkmer, mind taking a look at this issue as it has been labeled with an integration (honeywell) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `honeywell` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign honeywell` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

honeywell documentation honeywell source (message by IssueLinks)

mkmer commented 1 year ago

That is a password dialog, not sure why it isn't showing the "password" text. Try entering your password.

I noticed the login failure today for the first time in my setup - this may be related to web site issues, or a bug in the login flow when the keys expire. Let's see if it happens again in a given time frame.

OGTK423 commented 1 year ago

I suddenly am currently encountering the same issue. I do not receive a password prompt.

mkmer commented 1 year ago

The blank password is addressed in PR#88733. (It's in and will be released in 2023.3.0) The rauth being triggered when your password has not actually changed should be addressed in PR#88970 (Let's hope it gets in 2023.3.0) Honeywell is having troubles over the last 12 hours which helped me to figure out why we were occasionally asking for a reauth when the password was still good.

OGTK423 commented 1 year ago

I'm currently only receiving the

Retrying setup: Failed to initialize the Honeywell client: Connection error: maybe you have exceeded the API rate limit?

message. Reloading the integration once or twice will sometimes fix it but a few minutes later the error message returns or changes to

Failed to set up. Check the logs.

I don't receive a prompt to reauthenticate like the OP. Would this just be due to issues on Honeywell's end?

mkmer commented 1 year ago

The site is having issues today - many requests are returning 503/401. Just checked my logs and every minute or two I'm getting 503 (that's about 25-50% of the updates). The log entry was left over from the porting process, API rate limit is not "really" an API limit, it just a 503 or 401 error when we try to refresh. 2023.3.0 will clean up the error reporting and keep trying to get data (no need to reload).

aptalca commented 1 year ago

I'm on HA Core 2023.3.5 and this is still happening. I get the notification about the plugin needing reconfiguration about once a week and I'm presented with a text input box. I enter the same password, works again.

We all know that Honeywell's api really sucks and is not very reliable. For some reason, this plugin thinks that the password is no longer good based on the returned response from the api, but it is. Perhaps the timeout or the retries need to be increased?

Thanks

mkmer commented 1 year ago

https://github.com/home-assistant/core/pull/88970 This should have fixed the issue - I found that when the site goes down on the backend, the API was declaring it a password error. Must be I missed something?!? It hasn't been a problem on my installation... it's possible I haven't left it running long enough. Try turning on full logging for the integration so we can see what path is giving us issues.

StevenDTX commented 1 year ago

I'm on HA Core 2023.3.5 and this is still happening. I get the notification about the plugin needing reconfiguration about once a week and I'm presented with a text input box. I enter the same password, works again.

We all know that Honeywell's api really sucks and is not very reliable. For some reason, this plugin thinks that the password is no longer good based on the returned response from the api, but it is. Perhaps the timeout or the retries need to be increased?

Thanks

I had to delete and re-create the integration to get my devices to connect again.

mkmer commented 1 year ago

@stevenDTX Please open a new issue - this issue is for API rate limit

StevenDTX commented 1 year ago

@StevenDTX Please open a new issue - this issue is for API rate limit

Yes, that's the problem I was having. Even after the update, I could not get the error to clear until I removed the integration and added it again.

aptalca commented 1 year ago

Sure enough a couple of days in and it failed again. But the difference is, now it doesn't give me the reconfigure option, but it shows up in a red box and says Failed to set up Check the logs. Doing a manual reload made it work again.

Here are the individual log entries (I had forgotten to turn off on debug logging at the time)

1)

Logger: homeassistant.components.climate
Source: helpers/entity_platform.py:822
Integration: Climate (documentation, issues)
First occurred: March 26, 2023 at 9:30:55 PM (2 occurrences)
Last logged: March 28, 2023 at 9:13:30 PM

Updating honeywell climate took longer than the scheduled update interval 0:00:30

2)

Logger: homeassistant.helpers.entity
Source: components/honeywell/climate.py:423
First occurred: March 26, 2023 at 9:30:55 PM (3 occurrences)
Last logged: March 28, 2023 at 9:13:30 PM

Update for climate.main_thermostat fails
Update for climate.sunken_room fails
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/homeassistant/helpers/entity.py", line 541, in async_update_ha_state
    await self.async_device_update()
  File "/usr/lib/python3.10/site-packages/homeassistant/helpers/entity.py", line 747, in async_device_update
    await task
  File "/usr/lib/python3.10/site-packages/homeassistant/components/honeywell/climate.py", line 423, in async_update
    await self._device.refresh()
  File "/usr/lib/python3.10/site-packages/aiosomecomfort/device.py", line 45, in refresh
    data = await self._client._get_thermostat_data(self.deviceid)
  File "/usr/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 167, in _get_thermostat_data
    return await self._get_json(url)
  File "/usr/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 111, in _request_json
    resp: aiohttp.ClientResponse = await getattr(self._session, method)(
  File "/usr/lib/python3.10/site-packages/aiohttp/client.py", line 467, in _request
    with timer:
  File "/usr/lib/python3.10/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError

3)

Logger: somecomfort
Source: components/honeywell/climate.py:423
First occurred: March 26, 2023 at 8:44:54 AM (6 occurrences)
Last logged: 12:39:32 AM

401 Error at update (Key expired?).
API returned 200 from /portal/Device/CheckDataSession/7057944 request
API returned 500 from /portal/Device/CheckDataSession/7057944 request

4)

Logger: somecomfort
Source: components/honeywell/climate.py:430
First occurred: 12:39:33 AM (1 occurrences)
Last logged: 12:39:33 AM

Login as redacted@gmail.com failed

5)

Logger: homeassistant.config_entries
Source: components/honeywell/__init__.py:58
First occurred: 12:39:49 AM (1 occurrences)
Last logged: 12:39:49 AM

Error setting up entry honeywell for honeywell
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/homeassistant/config_entries.py", line 383, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/usr/lib/python3.10/site-packages/homeassistant/components/honeywell/__init__.py", line 58, in async_setup_entry
    await client.discover()
  File "/usr/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 194, in discover
    raw_locations = await self._get_locations()
  File "/usr/lib/python3.10/site-packages/aiosomecomfort/__init__.py", line 153, in _get_locations
    resp = await self._session.post(url, params=params, headers=self._headers)
  File "/usr/lib/python3.10/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/usr/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 899, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer

Not sure which one caused the integration to totally fail as I get a lot of these regularly (thanks to their awful api)

I now turned debug logging on and will report back next time it fails again

By the way, this is all on 2023.3.6 Core install in docker

Thanks

aptalca commented 1 year ago

@mkmer 2 days later, issue happened again, this time with asking for reconfiguration and it shows: Failed to set up: Incorrect Password

Relevant log entries:

Logger: somecomfort
Source: components/honeywell/climate.py:423
First occurred: March 30, 2023 at 12:43:45 AM (5 occurrences)
Last logged: 5:04:41 AM

401 Error at update (Key expired?).
API returned 200 from /portal/Device/CheckDataSession/7057944 request
API returned 200 from /portal/Device/CheckDataSession/6955933 request

and

Logger: somecomfort
Source: components/honeywell/climate.py:430
First occurred: 5:04:41 AM (1 occurrences)
Last logged: 5:04:41 AM

Login as redacted@gmail.com failed

Also the usual

Logger: homeassistant.helpers.entity
Source: components/honeywell/climate.py:430
First occurred: March 30, 2023 at 9:19:15 PM (6 occurrences)
Last logged: March 30, 2023 at 9:20:15 PM

Update for climate.main_thermostat fails
Update for climate.sunken_room fails
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/usr/lib/python3.10/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/usr/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 863, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/homeassistant/helpers/entity.py", line 541, in async_update_ha_state
    await self.async_device_update()
  File "/usr/lib/python3.10/site-packages/homeassistant/helpers/entity.py", line 747, in async_device_update
    await task
  File "/usr/lib/python3.10/site-packages/homeassistant/components/honeywell/climate.py", line 430, in async_update
    await self._data.client.login()
  File "/usr/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/usr/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/lib/python3.10/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.mytotalconnectcomfort.com:443 ssl:default [Try again]

and

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:541
First occurred: March 30, 2023 at 12:43:43 AM (7 occurrences)
Last logged: March 30, 2023 at 9:20:15 PM

Update of climate.sunken_room is taking over 10 seconds
Update of climate.main_thermostat is taking over 10 seconds

I do have 2 thermostats and maybe that's why I'm hitting these issues more frequently than others. Not sure.

When I hit Reconfigure, I get the blank password prompt as before (I thought that behavior was changed and that the current password should be populated).

Thanks

mkmer commented 1 year ago

This is going to be a difficult one. The web site back end is going down, then when we attempt to connect it returns an auth error.
If we just assume it's not an auth error and keep trying, you will need to remove and reinstall the integration to change the password... OR we have to enter the password when this occurs.

Please open a new ticket with this info as it's not related to the API limit, rather a different problem with handling of website being off line.