search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.49k stars 30.33k forks source link

Client unable to keep up with pending messages #81293

Closed smartmatic closed 1 year ago

smartmatic commented 1 year ago

The problem

I am getting the below error in my logs and don't know how to solve it as there are not many informations included.

What version of Home Assistant Core has the issue?

2022.10.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

WebSocket API

Link to integration documentation on our website

https://www.home-assistant.io/integrations/websocket_api

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:157
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 10:16:02 AM (1 occurrences)
Last logged: 10:16:02 AM

[546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds

2022-10-29 17:46:56.245 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547822567184] Received unknown command: mobile_app/push_notification_channel
2022-10-29 17:47:13.085 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547822567184] Received unknown command: mobile_app/push_notification_channel
2022-10-29 17:47:32.970 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547822567184] Received unknown command: mobile_app/push_notification_channel
2022-10-29 17:47:53.372 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547822567184] Received unknown command: mobile_app/push_notification_channel
2022-10-29 17:48:15.367 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547822567184] Received unknown command: mobile_app/push_notification_channel
2022-10-29 18:02:30.922 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [546897200800] Connection closed by client
2022-10-29 18:03:25.356 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [547020457584] Connection closed by client
client
2022-10-30 22:59:46.725 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [546200020800] Connection closed by client
2022-10-30 22:59:47.713 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543921260720] Connection closed by client
2022-10-30 23:06:05.183 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:07:25.185 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:08:48.254 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:10:11.328 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:11:34.399 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:12:57.520 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:13:36.367 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543921237120] Connection closed by client
2022-10-30 23:13:36.368 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543921228000] Connection closed by client
2022-10-30 23:14:20.608 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:15:44.664 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:17:07.907 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:18:30.975 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:19:54.048 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:21:05.940 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [546200018832] Connection closed by client
2022-10-30 23:21:17.117 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:22:40.190 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:24:03.323 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:25:26.400 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:26:49.469 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:28:12.544 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:29:35.619 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:29:42.828 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543930811584] Connection closed by client
2022-10-30 23:30:58.721 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:32:21.823 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:33:44.897 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:35:07.967 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:36:31.056 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:37:35.542 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543934808656] Connection closed by client
2022-10-30 23:37:35.546 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [543934818784] Connection closed by client
2022-10-30 23:37:54.285 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:39:17.348 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:40:40.415 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:42:03.523 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:43:26.591 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:44:49.663 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:46:12.735 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:47:35.805 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:48:58.943 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:50:22.016 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:51:45.090 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:53:08.159 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:54:31.231 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:55:54.302 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:57:17.374 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:58:29.408 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-30 23:58:40.447 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-31 00:00:03.585 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-31 00:01:26.653 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2022-10-31 00:02:49.727 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546992482976] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (websocket_api) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `websocket_api` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Change the title of the issue. - `@home-assistant unassign websocket_api` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

websocket_api documentation websocket_api source (message by IssueLinks)

erkr commented 1 year ago

Hi, this is reported more often. Have a look here: https://github.com/home-assistant/core/issues/68030 Long message short. The issue is caused by either high cpu load or high IO rates. Bad news is that it Can be caused everywhere. Most official HA code is cleaned up. So start with disabling custom integrations to check if one of them is causing this.

redwireless commented 1 year ago

Same here, began about a month ago when we decided to implement the 'time & date' integration AND while we're logged in to HA remotely. We deleted the 'time & date' integration from the config.yaml, restarted, even restored bkup before change - no matter. When we log in remotely to HA the CPU will rise to 49-51% consistently and will get the same error 'Client exceeded max pending messages'. Logging out and back in resolves (temporarily) the issue, bringing the CPU to 1%...until it happens again. It doesn't impede HA service, but bogs session down and drives VM/PC resources to max: Home Assistant 2022.10.5 Supervisor 2022.10.0 Operating System 9.3 Frontend 20221010.0 - latest Running on VM-Acer AMD SFF (Small Form Factor; similar to a NUC)🔴