Honeywell Total Connect Comfort (US) Being "Rate-Limited"

ziptbm commented 3 years ago

The problem

After taking 2021.8 added Honeywell Total Connect Comfort (US) support through the UI, I removed the YAML for my thermostat. However, I am noticing it working right after a reboot, but then it times out shortly after with an error indicating that I am being "rate-limited." This was not an apparent issue in 2021.7, but it's continued into 2021.9. Now after a day, my thermostat is unresponsive via Home Assistant until my next reboot.

What is version of Home Assistant Core has the issue?

2021.9.4

What was the last working version of Home Assistant Core?

2021.7

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Honeywell Total Connect Comfort (US)

Link to integration documentation on our website

https://www.home-assistant.io/integrations/honeywell/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.helpers.entity
Source: components/honeywell/__init__.py:127
First occurred: September 7, 2021, 05:47:26 (590 occurrences)
Last logged: 11:19:16

Update for climate.thermostat fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/honeywell/__init__.py", line 116, in update
    await self._hass.async_add_executor_job(self._refresh_devices)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/honeywell/__init__.py", line 108, in _refresh_devices
    device.refresh()
  File "/usr/local/lib/python3.9/site-packages/somecomfort/client.py", line 87, in refresh
    data = self._client._get_thermostat_data(self.deviceid)
  File "/usr/local/lib/python3.9/site-packages/somecomfort/client.py", line 468, in _get_thermostat_data
    return self._get_json(url)
  File "/usr/local/lib/python3.9/site-packages/somecomfort/client.py", line 444, in _get_json
    return self._request_json('get', *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/somecomfort/client.py", line 436, in _request_json
    raise APIRateLimited()
somecomfort.client.APIRateLimited: You are being rate-limited. Try waiting a bit.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 446, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 649, in async_device_update
    await task
  File "/usr/src/homeassistant/homeassistant/components/honeywell/climate.py", line 387, in async_update
    await self._data.update()
  File "/usr/src/homeassistant/homeassistant/components/honeywell/__init__.py", line 127, in update
    result = await self._hass.async_add_executor_job(self._retry())
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
TypeError: 'coroutine' object is not callable

Additional information

No response

probot-home-assistant[bot] commented 3 years ago

honeywell documentation honeywell source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 3 years ago

Hey there @rdfurman, mind taking a look at this issue as it has been labeled with an integration (honeywell) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

rdfurman commented 3 years ago

Duplicate of https://github.com/home-assistant/core/issues/55095

meyerrj commented 3 years ago

This was supposedly fixed with this PR: https://github.com/home-assistant/core/pull/55304 and it was included in today's release: https://www.home-assistant.io/blog/2021/09/01/release-20219/#release-202195---september-8

LordByronX commented 3 years ago

updated to 2021.9.5 and the rate limit is still occurring. seems to be worse, my thermostats were unresponsive after a few hours and had the rate limit error in the logs.

rdfurman commented 3 years ago

updated to 2021.9.5 and the rate limit is still occurring. seems to be worse, my thermostats were unresponsive after a few hours and had the rate limit error in the logs.

Can you post the error you're seeing in the logs please?

LordByronX commented 3 years ago

Logger: homeassistant.components.honeywell.const Source: components/honeywell/init.py:135 Integration: Honeywell Total Connect Comfort (US) (documentation, issues) First occurred: 11:45:12 AM (3 occurrences) Last logged: 1:45:11 PM

SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Read timed out. (read timeout=30)

rdfurman commented 3 years ago

Logger: homeassistant.components.honeywell.const Source: components/honeywell/init.py:135 Integration: Honeywell Total Connect Comfort (US) (documentation, issues) First occurred: 11:45:12 AM (3 occurrences) Last logged: 1:45:11 PM

SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Read timed out. (read timeout=30)

That isn't a rate limit error. A time out would make me think you just need to try again. Are you seeing this wire in the log frequently?

ADobin commented 3 years ago

I'm getting errors with the Honeywell integration after upgrading to 2021.9.5, although it's not the same rate limit as before, but it appears to consistently hit ~2 hours after starting Home Assistant. After this shows up the entity never updates again.

Logger: homeassistant.components.honeywell.const
Source: components/honeywell/__init__.py:135
Integration: Honeywell Total Connect Comfort (US) (documentation, issues)
First occurred: 4:06:33 PM (1 occurrences)
Last logged: 4:06:33 PM

SomeComfort update failed, Retrying - Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

meyerrj commented 3 years ago

It's quite possible that this could be a server-side issue with Honewell's services. I did encounter some similar disconnect errors in my logs earlier today, and I had some issues with their android app connecting as well.

blackburn42330 commented 3 years ago

I’m getting an error after 9.5 as well. Before Honeywell was an integration and I just had it installed as a Yaml code in my config, it worked great. Since it has been a listed integration, I’ve struggled getting it to work without frequent reboots.

Logger: homeassistant.components.honeywell.const Source: components/honeywell/init.py:135 Integration: Honeywell Total Connect Comfort (US) (documentation, issues) First occurred: September 10, 2021, 11:26:32 PM (4 occurrences) Last logged: 2:46:34 AM

SomeComfort update failed, Retrying - Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

LordByronX commented 3 years ago

Here is my latest error. If I restart HA, the integration works for about 2 hours and then shuts down with errors. Before this version is was working for about 4 to 6 hours before shutting down.

Logger: homeassistant.components.honeywell.const Source: components/honeywell/init.py:135 Integration: Honeywell Total Connect Comfort (US) (documentation, issues) First occurred: September 10, 2021, 2:52:03 AM (25 occurrences) Last logged: 8:50:10 AM

SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Max retries exceeded with url: /portal?UserName=%40gmail.com&Password=false&timeOffset=480 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9ab37699a0>, 'Connection to www.mytotalconnectcomfort.com timed out. (connect timeout=30)')) SomeComfort update failed, Retrying - Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Max retries exceeded with url: /portal?UserName=40gmail.com&Password=false&timeOffset=480 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9a9b7709a0>, 'Connection to www.mytotalconnectcomfort.com timed out. (connect timeout=30)')) SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Max retries exceeded with url: /portal?UserName=%40gmail.com&Password=false&timeOffset=480 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9aa1b097c0>, 'Connection to www.mytotalconnectcomfort.com timed out. (connect timeout=30)')) SomeComfort update failed, Retrying - Error: HTTPSConnectionPool(host='www.mytotalconnectcomfort.com', port=443): Max retries exceeded with url: /portal?UserName=40gmail.com&Password=false&timeOffset=480 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9aa1390550>, 'Connection to www.mytotalconnectcomfort.com timed out. (connect timeout=30)'))

meyerrj commented 3 years ago

@LazyFoxFarms FYI, if you haven't done so already, you may want to change your credentials ASAP...

If someone has the access, they should probably scrub the details in the comment above...

rdfurman commented 3 years ago

timed out. (connect timeout=30)'))

I was seeing errors logging into the TCC app directly yesterday. Could have been a TCC service issue since the error you saw was a timeout.

cpaplaczyk commented 3 years ago

Receiving the same rate limit issue. If you adjust the following code to a higher limit this may resolve your issue. Please note with this change the requests from updates directly on the Total Comfort App will take some additional time to update within your Home Assistant UI.

Adjust: MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=300) to MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=1800)

File location: https://github.com/home-assistant/core/blob/3aed58f825c310826d4ea16b785f34d49853d221/homeassistant/components/honeywell/__init__.py

Within a Docker Home Assistant deployment this file can be found in this directory: You first need to connect to the docker container from your docker host: docker exec -it <name of your home assistant container> /bin/bash Then edit the file: vi /usr/src/homeassistant/homeassistant/components/honeywell/__init__.py

There seems to be an issue with Home Assistants rate limit functionality but I have not had time to review/revert it, this may be a good temporary solution.

grahamr commented 3 years ago

the rate limit is 5 minutes, I found in my YAML testing setting to 340ish seconds for refresh would not get me locked, but being right at 300 would eventually due to retrying on a failure.

https://developer.honeywellhome.com/faqs/what-rate-limit-api

cpaplaczyk commented 3 years ago

the rate limit is 5 minutes, I found in my YAML testing setting to 340ish seconds for refresh would not get me locked, but being right at 300 would eventually due to retrying on a failure.

https://developer.honeywellhome.com/faqs/what-rate-limit-api

Good find!

rdfurman commented 3 years ago

the rate limit is 5 minutes, I found in my YAML testing setting to 340ish seconds for refresh would not get me locked, but being right at 300 would eventually due to retrying on a failure.

https://developer.honeywellhome.com/faqs/what-rate-limit-api

I know Honeywell have a few different API services (rosideo, maybe others?). Are you sure this is the same for the TCC US API?

If it is I'll update to 330 or something like that. 👍

chanba commented 3 years ago

@rdfurman Have you reached out to their developer relations email on that page? Maybe they can offer higher frequency for HA integration users? developerinfo@resideo.com

blackburn42330 commented 3 years ago

I personally don’t care if it’s every six minutes. I just want it to work reliably.

bakerkj commented 3 years ago

II am also seeing rate limiting while running 2021.9.6. I will try changing: MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=300) to 360 seconds and see if it works any better.

blackburn42330 commented 3 years ago

I’m not running in docker and can’t figure out how to change this myself. Is it possible?

LordByronX commented 3 years ago

Can you add the rate limit setting into the integration setup so we can set our own rate? I think I had mine set at 500 in the old yaml config. I am still getting the rate limit errors - the timeout is because the rate limit.

thanks for doing all this! Your work is appreciated!

bakerkj commented 3 years ago

II am also seeing rate limiting while running 2021.9.6. I will try changing: MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=300) to 360 seconds and see if it works any better.

I have 5 thermostats. This did not work for me. After two days I am once again getting a rate limiting error.

rdfurman commented 3 years ago

II am also seeing rate limiting while running 2021.9.6. I will try changing: MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=300) to 360 seconds and see if it works any better.

I have 5 thermostats. This did not work for me. After two days I am once again getting a rate limiting error.

"rate limiting error" is a red herring. Here is a thread for the real issue that we're looking into: https://github.com/home-assistant/core/issues/56034?notification_referrer_id=MDE4Ok5vdGlmaWNhdGlvblRocmVhZDI0MDAwNzkyNjI6MjU0NzQ0OA%3D%3D#issuecomment-925479334

We can close this issue.

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core