search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.48k stars 30.7k forks source link

ERROR: "Client unable to keep up with pending messages. Stayed over 512 for 5 seconds" at the system log #40443

Closed dpositive closed 3 years ago

dpositive commented 4 years ago

The problem

ERROR: "Client unable to keep up with pending messages. Stayed over 512 for 5 seconds" appears at the system log constantly

Environment

subject info
arch armv7
chassis  
dev false
docker true
docker_version 19.03.12
hassio true
host_os Raspbian GNU/Linux 10 (buster)
installation_type Home Assistant Supervised
os_name Linux
os_version 5.4.51-v7+
python_version 3.8.5
supervisor 245
timezone Europe/Kiev
version 0.115.2
virtualenv false

Problem-relevant configuration.yaml

homeassistant:
  name: "aaa"
  latitude: !secret latitude_coord
  longitude: !secret longitude_coord
  elevation: 179
  unit_system: metric
  time_zone: Europe/Kiev
  customize: !include includes/customize.yaml
  external_url: !secret external_url
  internal_url: !secret internal_url
  packages: !include_dir_merge_named includes/packages
  whitelist_external_dirs: 
    - /config  

config:

logger:
  default: info

frontend:
  themes: !include_dir_merge_named themes

map:  

mobile_app:

sun:

system_health:

updater:

http:
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

lovelace:
  mode: yaml
  resources: !include includes/resources.yaml

history:

hacs:
  token: !secret HACS_github
  appdaemon: true
  python_script: true
  theme: true

google_assistant:
  project_id: !secret google_assistant_project_id
  exposed_domains:
   - switch
   - light
   - climate
   - mediaplayer
   - input_boolean
   - scene
   - script
   - sensor

tts:
  - platform: google_translate

mqtt:
  broker: core-mosquitto
  discovery: true
  discovery_prefix: homeassistant  
  username: !secret mqtt_username
  password: !secret mqtt_password

smartir:

timer: !include includes/timer.yaml
input_boolean: !include includes/input_boolean.yaml
input_number: !include includes/input_number.yaml
input_select: !include includes/input_select.yaml
group: !include includes/groups.yaml
scene: !include includes/scenes.yaml
recorder: !include includes/recorder.yaml
climate: !include includes/climate.yaml
zone: !include includes/zone.yaml
person: !include includes/person.yaml
telegram_bot: !include includes/telegram_bot.yaml
notify: !include includes/notify.yaml
remote: !include includes/remote.yaml

sensor: !include_dir_merge_list includes/sensor
automation: !include_dir_merge_list includes/automation
binary_sensor: !include_dir_merge_list includes/bin_sensor
script: !include_dir_merge_named includes/scripts
switch: !include_dir_merge_list includes/switches

Traceback/Error logs

Logger: homeassistant.components.websocket_api.http.connection.1844975792
Source: components/websocket_api/http.py:138
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 00:37:19 (1 occurrences)
Last logged: 00:37:19
Client unable to keep up with pending messages. Stayed over 512 for 5 seconds

Additional information

dzianiwonsacze commented 3 years ago

OK, it may be too early to call it a victory, but I would suggest checking CPU load - mine was nearly 100%, likely because of the heat generated by closed cabinet the server was in. When I opened it in the morning and let more air in, the load went down to around 30% and there are no more errors. Zigbee performance went up also.

picotrain77 commented 3 years ago

My CPU runs at about 30% and never much higher.

I seem to get this error at almost exactly the same time once every morning and then nothing until the following morning

balloob commented 3 years ago

I wouldn't be surprised if you have something running around that time (a backup?) that peaks the CPU to 100%.

aceindy commented 3 years ago

2021-07-27 21:11:40 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [156746359965678] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds core-2021.7.4 image Running on HyperV on W10 with i7 proc...never noticed heavy cpu load, but error appears a couple of time a day,

vistalba commented 3 years ago

I've the same issue. Running Homassistant OS on ESXi and having also low RAM/CPU usage.

johnekarlsson commented 3 years ago

Same here. Also, my hue lights becomes unavailable shortly after. I suspect that the two problems are related.

Flodu31 commented 3 years ago

Hello, Same problem, a lot of messages:

2021-09-09 06:31:30 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client exceeded max pending messages [2]: 2048 2021-09-09 06:31:31 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client exceeded max pending messages [2]: 2048 2021-09-09 06:31:31 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client exceeded max pending messages [2]: 2048 2021-09-09 06:31:31 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client exceeded max pending messages [2]: 2048 2021-09-09 06:31:31 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client exceeded max pending messages [2]: 2048 2021-09-09 06:31:32 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform rest is taking over 10 seconds. 2021-09-09 06:31:35 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2845511296] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds

Any idea? Thanks. Florent

bkbartk commented 3 years ago

the issue is closed, but I think it should be opened again. 2021-09-20 19:13:14 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [1746431136] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds 2021-09-20 19:13:20 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [1746431136] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds I have this issue too,

I see it is suggested this is a client browser issue, but I think the browser should never be able to crash the server.

hellcry37 commented 3 years ago

up for reopen the issue

LAKY911 commented 3 years ago

Happened to me today.

OzGav commented 3 years ago

Reading through this whole thread I think it might be related to use of the developer tools on my iPad. When I start typing the lag to update the list of entities is substantial. I say this based on what @kmalinich and @bdraco wrote back on the 23-29 April

kmalinich commented 3 years ago

Reading through this whole thread I think it might be related to use of the developer tools on my iPad. When I start typing the lag to update the list of entities is substantial. I say this based on what @kmalinich and @bdraco wrote back on the 23-29 April

I experience the same thing on in the States page in Developer Tools on my Android phone. It also occurs on my macOS desktop, but only when opening Developer Tools from within the desktop app, not in Chrome.

bkbartk commented 3 years ago

I increased my swap file. This seemed to stop the issue. Next to that I also replaced the charger. So it can be both. But I think increasing the swap file will do the trick.

kmalinich commented 3 years ago

I don't think it's a server side issue, nor is it crashing anything. The issue occurs on hardware with much more resources than is required. If anything, we might want to look at issues in the mobile app repo. For me, accessing the HA web UI (specifically developer tools->states, or editing a card and typing in the entity field), via the HA mobile app is the only thing that makes the issue occur.

benblb commented 3 years ago

This is happening for me as well. i5 with 8GB of RAM which is in line with the theory from @kmalinich .

julianrinaldi commented 3 years ago

I was having this error previously, along with a warning about too many open files. I run Home Assistant Core on a Mac. I changed the ulimit by entering the command "ulimit -n 65536 200000" in my python environment, and this was definitely responsible for stopping the too many open files issue, but a while later I noticed that I was no longer get the pending messages error. So it might be worth a try to increase the ulimit and see if that helps.

wtadler commented 3 years ago

This is still happening for me on core-2021.11.1, on a Raspberry Pi 4B.

hellcry37 commented 2 years ago

Just trowing this out there but how many of you have this installed:

https://www.home-assistant.io/integrations/bluetooth_tracker/

For me this was a real problem and as soon as I've remove it i cut HA startup time to less then 33% from over 1-2 minutes to a few seconds.

vistalba commented 2 years ago

Just trowing this out there but how many of you have this installed:

https://www.home-assistant.io/integrations/bluetooth_tracker/

For me this was a real problem and as soon as I've remove it i cut HA startup time to less then 33% from over 1-2 minutes to a few seconds.

I do not have this installed but having the issue too.

Qpernicus commented 2 years ago

In my case it seems to be directly linked to 1 of 2 things.

1) I made an extra dashboard with HomeKit-panel card (custom). And added this card on a few places in my other dashboards (5 visible + 4 'hidden' from sidebar, added the card to 2).

2) When I push a light button (based on above homekitpanelcard ) I get the message about the 512 messages.

So in my case it seems it could either be: to much dashboards or the HomeKit panel card. But perhaps it all comes down to just to much changes to keep up. Most of the cards look like this (except one dashboard, that is a panelview with one big homekit-panel-card

type: custom:homekit-card enableColumns: true rows:

  • row: 1 columns:
    • column: 2 entities:
      • title: Kamerverlichting entities:
        • entity: light.lampen_tv name: Televisiespots
        • entity: light.lampen_bank name: Zithoekspots
        • entity: sensor.vibration_sensor_2_temperature name: Temperatuur icon: mdi:thermometer hideState: true
    • column: 2 entities:
      • title: Keukenverlichting entities:
        • entity: light.lampen_aanrecht name: Aanrechtspots
        • entity: light.lampen_keuken name: Plafondspots
        • entity: sensor.motion_sensor_temperature name: Temperatuur icon: mdi:thermometer hideState: true
benblb commented 2 years ago
  1. I made an extra dashboard with HomeKit-panel card (custom). And added this card on a few places in my other dashboards (5 visible + 4 'hidden' from sidebar, added the card to 2).
  2. When I push a light button (based on above homekitpanelcard ) I get the message about the 512 messages.

I also had an extra dashboard with Custom Button Card and a light. Just deleted the second dash since it was simply for testing.

Qpernicus commented 2 years ago

I just removed 1 visible and 4 'invisible ' dashboards. Still have 2 with the HomeKit (panel) card en still get the error. I believe this card to be the culprit but don't understand why.

Dominik-1980 commented 2 years ago

I have the same problem since yesterday. I have installed Nuki Lock over HACS. It uses callbacks. Perhaps it is now too much http traffic?

ildar170975 commented 2 years ago

Same with 2021.11.5 Win10x64, Chrome HA in Docker (RPi)

bcutter commented 2 years ago

Still on 2022.6.7:

Massively exists:

Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
First occurred: 00:52:50 (9061 occurrences)
Client exceeded max pending messages [2]: 2048
First occurred: 07:29:00 (7484 occurrences)

Floods HA log pretty quick.

bdraco commented 2 years ago

This wasn't fixed until 2022.9.x

bcutter commented 2 years ago

So why is it closed.

bdraco commented 2 years ago

Because we close issues when we merge fixes