search

home-assistant / frontend

:lollipop: Frontend for Home Assistant
https://demo.home-assistant.io
Other
4.06k stars 2.77k forks source link

Memory leak when keeping the dashboard open on a tablet or mobile (narrow) device (fixed in 2023.7) #16952

Closed VinceRMP closed 1 year ago

VinceRMP commented 1 year ago

The problem

The memory leak seems to have started after updating HAOS to 10.2 or core 2023.6 beta (coming from a dev build in the early stage of 2023.5, so it could also be related to a late 2023.5 specific version). What I have noticed is that my memory increase is caused by my tablet which shows the dashboard 24/7. I can reproduce the start of the memory leak and stop it aswell after some testing. Bdraco had a look in my (profiler) logs and he said there are no obvious task leaks in the them, so I'm not sure if it is an issue in core or frontend.

image At the first drop in memory I did a restart of Home Assistant, but didn't refresh the webapp running on my tablet. This causes the memory to increase as you can see. Around 08:00 I restarted the webapp on the tablet. The memory stays the same, so no memory leak visible. Around 09:00 I did another restart of Home Assistant without restarting the webapp on the tablet and there is the memory leak again.

This is also reported in the logs: [139928754006096] Tablet from (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":22632,"type":"result","success":true,"result":null} [139928813456528] Tablet from (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":20399,"type":"result","success":true,"result":null} [139928532840656] Tablet from (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":20538,"type":"result","success":true,"result":null} [139928688859152] Tablet from (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":70314,"type":"result","success":true,"result":null} [139928776142096] Tablet from (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":30737,"type":"result","success":true,"result":null}

What version of Home Assistant Core has the issue?

core-2023.6.2

What was the last working version of Home Assistant Core?

core-2023.5.0?

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

backcountrymountains commented 1 year ago

I, too, have noticed memory leak errors and am running Fully Kiosk on several tablet and tablet-like devices with a similar error when I restart HA and throughout the day:

    Line   210: 2023-06-16 18:19:12.853 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666562449744] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":8136,"type":"result","success":true,"result":null}
    Line   211: 2023-06-16 18:19:13.277 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666591085776] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":26455,"type":"result","success":true,"result":null}
    Line   335: 2023-06-16 18:19:28.081 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666532112016] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4489,"type":"result","success":true,"result":null}
    Line   356: 2023-06-16 18:19:29.583 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666514244880] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":6529,"type":"result","success":true,"result":null}
    Line   487: 2023-06-16 18:19:40.097 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666511210512] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":8015,"type":"result","success":true,"result":null}
    Line   488: 2023-06-16 18:19:40.334 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666518128208] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4489,"type":"result","success":true,"result":null}
    Line   697: 2023-06-16 18:19:47.954 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666504662224] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":23747,"type":"result","success":true,"result":null}
    Line   698: 2023-06-16 18:19:48.377 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666040378896] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4489,"type":"result","success":true,"result":null}
    Line   701: 2023-06-16 18:19:49.821 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666514743824] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":8132,"type":"event","event":{"type":"current","notifications":{}}}
    Line   778: 2023-06-16 18:19:57.576 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665850495632] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":7791,"type":"result","success":true,"result":null}
    Line   779: 2023-06-16 18:19:58.371 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665856221648] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4488,"type":"result","success":true,"result":null}
    Line   922: 2023-06-16 18:20:04.398 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665876643344] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":5985,"type":"result","success":true,"result":null}
    Line   923: 2023-06-16 18:20:04.707 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666544631568] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":10174,"type":"result","success":true,"result":null}
    Line   930: 2023-06-16 18:20:06.241 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665801291024] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":8075,"type":"result","success":true,"result":null}
    Line  1067: 2023-06-16 18:20:15.471 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665852045200] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":20954,"type":"result","success":true,"result":null}
    Line  1690: 2023-06-16 18:20:20.143 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665853887120] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":18500,"type":"result","success":true,"result":null}
    Line  1692: 2023-06-16 18:20:21.503 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665721494096] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4487,"type":"result","success":true,"result":null}
    Line  1852: 2023-06-16 18:20:29.257 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665853731920] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":10227,"type":"result","success":true,"result":null}
    Line  1854: 2023-06-16 18:20:33.468 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665807202384] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":20103,"type":"result","success":true,"result":null}
    Line  1860: 2023-06-16 18:20:39.314 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665863771216] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":20048,"type":"event","event":{"type":"current","notifications":{}}}
    Line  1923: 2023-06-16 18:20:46.832 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665883281232] user from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":11021,"type":"result","success":true,"result":null}
    Line  1924: 2023-06-16 18:20:47.180 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665881695696] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":9370,"type":"result","success":true,"result":null}
    Line  1994: 2023-06-16 18:20:57.684 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140666527163280] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":8732,"type":"event","event":{"type":"current","notifications":{}}}
    Line  1999: 2023-06-16 18:21:10.180 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140665706349712] user from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":15774,"type":"result","success":true,"result":null}

This happens after I restart HA and usually causes Fully Kiosk to crash and then I have to open up the app again. I'm also running https://github.com/j-a-n/lovelace-wallpanel on the tablets.

I think you might be on to something because I similarly reopened the Fully Kiosk on the tablets and saw a stabilization of memory use on the server.

I reverted to HAOS 9.5 and my memory use now increases more slowly, but still keeps going up.

@VinceRMP hat browser/app are you using on your tablet? Do you use BrowserMod?

Me:

## System Information

version | core-2023.6.1
-- | --
installation_type | Home Assistant OS
dev | false
hassio | true
docker | true
user | root
virtualenv | false
python_version | 3.11.3
os_name | Linux
os_version | 5.15.90
arch | x86_64
timezone | America/Denver
config_dir | /config

<details><summary>Home Assistant Community Store</summary>

GitHub API | ok
-- | --
GitHub Content | ok
GitHub Web | ok
GitHub API Calls Remaining | 4791
Installed Version | 1.31.0
Stage | running
Available Repositories | 1289
Downloaded Repositories | 32

</details>

<details><summary>Home Assistant Supervisor</summary>

host_os | Home Assistant OS 9.5
-- | --
update_channel | stable
supervisor_version | supervisor-2023.06.2
agent_version | 1.4.1
docker_version | 20.10.22
disk_total | 30.8 GB
disk_used | 10.7 GB
healthy | true
supported | true
board | ova
supervisor_api | ok
version_api | ok
installed_addons | Samba share (10.0.1), Mosquitto broker (6.2.1), RPC Shutdown (2.4), File editor (5.6.0), Log Viewer (0.15.1), Advanced SSH & Web Terminal (15.0.1), Terminal & SSH (9.7.0), Duck DNS (1.15.0), ESPHome (2021.12.3), Z-Wave JS (0.1.80), FTP (4.2.1), go2rtc (1.5.0), SQLite Web (3.8.0), Custom deps deployment (1.3.3)

</details>

<details><summary>Dashboards</summary>

dashboards | 5
-- | --
resources | 22
views | 22
error | /config/notexist.yaml not found
mode | yaml

</details>

<details><summary>Recorder</summary>

oldest_recorder_run | May 22, 2023 at 10:18 PM
-- | --
current_recorder_run | June 17, 2023 at 8:52 AM
estimated_db_size | 3300.61 MiB
database_engine | sqlite
database_version | 3.41.2

</details>
ondras12345 commented 1 year ago

I think I have the same issue. I'm using wallpanel as my web browser. I haven't been able to reliably trigger the bug, so I don't know if it can be reproduced with another web browser. The frontend gets stuck and Home Assistant core starts leaking memory until it is killed by OOM. Even if I catch it in time and refresh the webpage, the memory that had leaked is never released. image

2023-06-13 07:35:50.942 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2547740368] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":5285,"type":"result","success":true,"result":null}
2023-06-13 07:35:53.472 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2478294032] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4061,"type":"result","success":true,"result":null}
2023-06-13 07:35:55.466 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2475810928] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4810,"type":"result","success":true,"result":null}
2023-06-13 07:35:57.681 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2471888656] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4701,"type":"result","success":true,"result":null}
2023-06-13 07:35:59.193 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2456017584] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":6810,"type":"result","success":true,"result":null}
2023-06-13 07:36:00.438 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2448000720] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":5136,"type":"result","success":true,"result":null}
2023-06-13 07:36:02.439 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2443593360] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4839,"type":"result","success":true,"result":null}
2023-06-13 07:36:03.772 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2476205520] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4881,"type":"result","success":true,"result":null}
2023-06-13 07:36:05.206 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2429982960] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":5274,"type":"result","success":true,"result":null}
2023-06-13 07:36:06.453 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2421368176] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4114,"type":"result","success":true,"result":null}
2023-06-13 07:36:08.336 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2410362064] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4203,"type":"result","success":true,"result":null}
2023-06-13 07:36:09.564 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2408794480] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4777,"type":"result","success":true,"result":null}
2023-06-13 07:36:13.478 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('cell', 558592, 300752), ('tuple', 320590, 137367), ('function', 294192, 120042), ('WSMessage', 22926, 17155), ('dict', 64823, 1611), ('HassJob', 3610, 1032), ('list', 58172, 894), ('frozenset', 5590, 885), ('method', 7253, 616), ('set', 11837, 576), ('hamt_bitmap_node', 636, 331), ('TrackTemplate', 504, 295), ('TrackTemplateResultInfo', 496, 295), ('KeyedRateLimit', 496, 295), ('RenderInfo', 504, 295), ('TrackStates', 498, 295), ('_TrackStateChangeFiltered', 498, 295), ('Context', 2781, 281), ('hamt', 358, 175), ('_FilterableJob', 342, 132), ('TimerHandle', 368, 115), ('Task', 105, 49), ('deque', 1339, 36), ('Handle', 64, 33), ('coroutine', 607, 25), ('MemoryBIO', 58, 20), ('CIMultiDict', 171, 14), ('FlowControlDataQueue', 22, 11), ('WebSocketHandler', 21, 11), ('WebSocketAdapter', 21, 11), ('ActiveConnection', 21, 11), ('WebSocketResponse', 21, 11), ('WebSocketWriter', 22, 11), ('memoryview', 30, 10), ('managedbuffer', 30, 10), ('_SSLProtocolTransport', 29, 10), ('_SSLSocket', 39, 10), ('CancelOnDisconnectRequestHandler', 28, 10), ('HomeAssistantAccessLogger', 28, 10), ('SSLProtocol', 29, 10), ('SSLObject', 29, 10), ('TimeoutError', 17, 10), ('builtin_function_or_method', 9149, 9), ('RawRequestMessage', 28, 9), ('Future', 47, 9), ('CIMultiDictProxy', 145, 9), ('HomeAssistantRequest', 27, 9), ('StreamWriter', 27, 9), ('UrlMappingMatchInfo', 27, 9), ('SplitResult', 161, 8), ('URL', 34, 8), ('SimpleCookie', 30, 8), ('CancelledError', 2, 1), ('dict_valueiterator', 1, 1)]
2023-06-13 07:36:14.308 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2401347888] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3969,"type":"event","event":{"type":"current","notifications":{"profile_object_logging":{"message":"Object growth logging has started. See [the logs](/config/logs) to track the growth of new objects.","notification_id":"profile_object_logging","title":"Object growth logging started","created_at":"2023-06-12T22:04:07.099740+00:00"}}}}
...
2023-06-13 07:38:24.777 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('cell', 2775089, 2216497), ('tuple', 1311501, 990911), ('function', 1178809, 884617), ('WSMessage', 127876, 104950), ('dict', 77276, 12453), ('HassJob', 11359, 7749), ('list', 65517, 7345), ('frozenset', 12022, 6432), ('method', 11747, 4494), ('set', 16192, 4355), ('Context', 6932, 4151), ('TimerHandle', 4232, 3864), ('hamt_bitmap_node', 3315, 2679), ('RenderInfo', 2648, 2144), ('TrackStates', 2642, 2144), ('_TrackStateChangeFiltered', 2642, 2144), ('TrackTemplateResultInfo', 2640, 2144), ('KeyedRateLimit', 2640, 2144), ('TrackTemplate', 2648, 2144), ('hamt', 1776, 1418), ('_FilterableJob', 1278, 936), ('deque', 1644, 305), ('coroutine', 864, 257), ('ReferenceType', 22927, 250), ('Task', 341, 236), ('CIMultiDict', 321, 150), ('MemoryBIO', 204, 146), ('FlowControlDataQueue', 100, 78), ('WebSocketHandler', 99, 78), ('WebSocketAdapter', 99, 78), ('ActiveConnection', 99, 78), ('WebSocketResponse', 99, 78), ('WebSocketWriter', 100, 78), ('Future', 123, 76), ('SplitResult', 235, 74), ('URL', 108, 74), ('SimpleCookie', 104, 74), ('memoryview', 103, 73), ('managedbuffer', 103, 73), ('RawRequestMessage', 101, 73), ('CIMultiDictProxy', 218, 73), ('_SSLProtocolTransport', 102, 73), ('_SSLSocket', 112, 73), ('HomeAssistantRequest', 100, 73), ('StreamWriter', 100, 73), ('UrlMappingMatchInfo', 100, 73), ('CancelOnDisconnectRequestHandler', 101, 73), ('HomeAssistantAccessLogger', 101, 73), ('SSLProtocol', 102, 73), ('SSLObject', 102, 73), ('TimeoutError', 83, 66), ('State', 1946, 32), ('ReadOnlyDict', 4352, 25), ('builtin_function_or_method', 9173, 24), ('TaskStepMethWrapper', 25, 17), ('Condition', 65, 2), ('MQTTMessage', 3, 2), ('MQTTMessageInfo', 3, 2)]
...
2023-06-13 07:40:27.994 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('cell', 3653593, 878504), ('tuple', 1698039, 386538), ('function', 1529368, 350559), ('WSMessage', 163290, 35414), ('dict', 82281, 5005), ('HassJob', 14569, 3210), ('list', 68629, 3112), ('frozenset', 14674, 2652), ('method', 13579, 1832), ('set', 17960, 1768), ('hamt_bitmap_node', 4456, 1141), ('_TrackStateChangeFiltered', 3526, 884), ('TrackStates', 3526, 884), ('TrackTemplateResultInfo', 3524, 884), ('KeyedRateLimit', 3524, 884), ('TrackTemplate', 3532, 884), ('RenderInfo', 3532, 884), ('hamt', 2376, 600), ('_FilterableJob', 1638, 360), ('deque', 1770, 126), ('ReferenceType', 23052, 125), ('MemoryBIO', 274, 70), ('Task', 409, 68), ('CIMultiDict', 385, 64), ('coroutine', 914, 50), ('memoryview', 138, 35), ('managedbuffer', 138, 35), ('_SSLProtocolTransport', 137, 35), ('_SSLSocket', 147, 35), ('CancelOnDisconnectRequestHandler', 136, 35), ('HomeAssistantAccessLogger', 136, 35), ('SSLProtocol', 137, 35), ('SSLObject', 137, 35), ('RawRequestMessage', 135, 34), ('CIMultiDictProxy', 252, 34), ('HomeAssistantRequest', 134, 34), ('StreamWriter', 134, 34), ('UrlMappingMatchInfo', 134, 34), ('SplitResult', 268, 33), ('URL', 141, 33), ('SimpleCookie', 137, 33), ('Future', 155, 32), ('FlowControlDataQueue', 130, 30), ('WebSocketHandler', 129, 30), ('WebSocketAdapter', 129, 30), ('ActiveConnection', 129, 30), ('WebSocketResponse', 129, 30), ('WebSocketWriter', 130, 30), ('TimeoutError', 110, 27)]

It seems to leak WSMessage objects.

Even if this is a frontend bug, I think the backend shouldn't be this easy to DoS.

home-assistant[bot] commented 1 year ago

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (websocket_api) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `websocket_api` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign websocket_api` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

websocket_api documentation websocket_api source (message by IssueLinks)

bdraco commented 1 year ago

Just a guess but it looks like something is creating tons of websocket connections and than throwing them away. They get left open until the back pressure builds up enough to disconnect them and you get the Client unable to keep up with pending messages message

bdraco commented 1 year ago

https://github.com/home-assistant/core/pull/94780 will add some more debug logging to the websocket api which may give a better idea about what is going on here.

silvannn commented 1 year ago

I have similar problem 192.168.0.40 it is my PC with HASS.Agent installed. One day HA restart itself 6 times in a row. I turn everything off. Unplug power from Raspberry Pi with HA and reboot system. Stay stable for 3 days and again problem is back. Again restart itself 2 times in a row. Memory use on that point was above 90%.

[547747023568] Sylwester from 192.168.0.40 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":2205,"type":"result","success":true,"result":null} [547708380944] Sylwester from 192.168.0.40 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":2118,"type":"event","event":{"type":"current","notifications":{}}} [547975880848] Sylwester from 192.168.0.40 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3396,"type":"result","success":true,"result":null} [547972134224] Sylwester from 192.168.0.40 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":2626,"type":"event","event":{"type":"current","notifications":{}}} [548355635984] Sylwester from 192.168.0.40 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":2423,"type":"result","success":true,"result":null}

bdraco commented 1 year ago

There are some other issues in the queue for memory leaks with HASS.Agent

maybe related as well https://github.com/LAB02-Research/HASS.Agent/issues/270#

silvannn commented 1 year ago

I am not sure if HASS.Agent is guilty. I am using it 1 year now and I did not have this problem before I update to 2023.6. I think that has some connection with Chrome when I have dashboard on screen. 2023-06-17

bdraco commented 1 year ago

I am not sure if HASS.Agent is guilty. I am using it 1 year now and I did not have this problem before I update to 2023.6. I think that has some connection with Chrome when I have dashboard on screen. 2023-06-17

I doubt the hass.agent issue is the same problem. If I had to guess it’s something with python 3.11

ondras12345 commented 1 year ago

I have applied the patch from https://github.com/home-assistant/core/pull/94780, enabled debug logging, restarted home assistant and was lucky enough to trigger the problem immediately.

This is the first appearance of "Client unable to keep up ...":

...
2023-06-17 22:19:43.380 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3250}
2023-06-17 22:19:43.380 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3251}
2023-06-17 22:19:43.380 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3252}
2023-06-17 22:19:43.380 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3253}
2023-06-17 22:19:43.381 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3254}
2023-06-17 22:19:43.381 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3255}
2023-06-17 22:19:43.381 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3256}
2023-06-17 22:19:43.381 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3257}
2023-06-17 22:19:43.382 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3258}
2023-06-17 22:19:43.382 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3259}
2023-06-17 22:19:43.382 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3260}
2023-06-17 22:19:43.382 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3261}
2023-06-17 22:19:43.383 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3262}
2023-06-17 22:19:43.383 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3263}
2023-06-17 22:19:43.383 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3264}
2023-06-17 22:19:43.383 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3265}
2023-06-17 22:19:43.384 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3266}
2023-06-17 22:19:43.386 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3267}
2023-06-17 22:19:43.386 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3268}
2023-06-17 22:19:43.387 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3269}
2023-06-17 22:19:43.387 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3270}
2023-06-17 22:19:43.387 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3271}
2023-06-17 22:19:43.388 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3272}
2023-06-17 22:19:43.388 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3273}
2023-06-17 22:19:43.388 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3274}
2023-06-17 22:19:43.388 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3275}
2023-06-17 22:19:43.389 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3276}
2023-06-17 22:19:43.389 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3277}
2023-06-17 22:19:43.389 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Received {'type': 'persistent_notification/subscribe', 'id': 3278}
2023-06-17 22:19:43.390 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [2790511408] Hass Term S3 mini from 192.168.1.63 (Mozilla/5.0 (Linux; Android 11; Pixel 2 Build/RP1A.201005.004.A1; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.131 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":3278,"type":"result","success":true,"result":null}

There's thousands of those persistent_notification/subscribe messages.

backcountrymountains commented 1 year ago

So, I don't know what I'm doing, but it seems that when I run:

service: profiler.start_log_objects
data: {}

My memory use stabilizes and I don't get OOM errors. image

I left HA alone all night and it went from 40% to 60%. I rebooted at 8:52 am to test out the refresh tablet theory, but the memory usage kept increasing. At 11:29 am I started the profiler:

2023-06-17 08:52:36.894 INFO (MainThread) [homeassistant.setup] Setup of domain profiler took 0.0 seconds
2023-06-17 11:29:45.191 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('cell', 391361, 391361), ('tuple', 255769, 255769), ('function', 228973, 228973), ('dict', 75390, 75390), ('list', 70783, 70783), ('ReferenceType', 28474, 28474), ('uint8_t', 19680, 19680), ('NodeDictClass', 18962, 18962), ('set', 15980, 15980), ('type', 14546, 14546), ('ReadOnlyDict', 14380, 14380), ('method', 10953, 10953), ('builtin_function_or_method', 10421, 10421), ('getset_descriptor', 9487, 9487), ('AIOGitHubAPIRepositoryTreeContent', 8704, 8704), ('uint16_t', 6712, 6712), ('property', 6206, 6206), ('Context', 5987, 5987), ('Element', 5962, 5962), ('frozenset', 5466, 5466), ('ZCLAttributeDef', 5385, 5385), ('PosixPath', 5022, 5022), ('State', 4972, 4972), ('Schema', 4923, 4923), ('HassJob', 4823, 4823), ('module', 4382, 4382), ('ModuleSpec', 4378, 4378), ('NodeListClass', 4136, 4136), ('SourceFileLoader', 4131, 4131), ('uint1_t', 3451, 3451), ('wrapper_descriptor', 3218, 3218), ('member_descriptor', 3082, 3082), ('partial', 2874, 2874), ('method_descriptor', 2566, 2566), ('NWK', 2550, 2550), ('TemplateState', 2324, 2324), ('Sequence', 2276, 2276), ('cython_function_or_method', 2252, 2252), ('RegistryEntry', 2224, 2224), ('StructField', 2219, 2219), ('Event', 2018, 2018), ('Optional', 1989, 1989), ('_GenericAlias', 1864, 1864), ('_abc_data', 1854, 1854), ('classmethod', 1772, 1772), ('Field', 1737, 1737), ('ABCMeta', 1529, 1529), ('ForwardRef', 1455, 1455), ('staticmethod', 1383, 1383), ('_UnionGenericAlias', 1352, 1352), ('RepositoryData', 1289, 1289), ('RepositoryContent', 1289, 1289), ('RepositoryPath', 1289, 1289), ('Validate', 1289, 1289), ('RepositoryReleases', 1289, 1289), ('HacsManifest', 1289, 1289), ('PathMetadata', 1273, 1273), ('DistInfoDistribution', 1272, 1272), ('Required', 1268, 1268), ('Parameter', 1266, 1266), ('Any', 1227, 1227), ('ZCLCommandDef', 1156, 1156), ('deque', 1129, 1129), ('OrderedDict', 1114, 1114), ('All', 1075, 1075), ('ModelField', 1001, 1001), ('FieldI ...
2023-06-17 11:30:17.611 CRITICAL (SyncWorker_60) [homeassistant.components.profiler] Memory Growth: [('WSMessage', 7950, 7935), ('dict', 75491, 101), ('ReferenceType', 28555, 81), ('method', 10990, 37), ('Handle', 92, 24), ('deque', 1152, 23), ('InstanceState', 89, 16), ('States', 17, 16), ('set', 15995, 15), ('list', 70797, 14), ('MemoryBIO', 60, 14), ('TraceElement', 151, 13), ('partial', 2885, 11), ('socket', 99, 10), ('KeyedRef', 364, 8), ('TransportSocket', 66, 8), ('uint8_t', 19688, 8), ('_SelectorSocketTransport', 47, 8), ('CIMultiDict', 58, 8), ('memoryview', 31, 7), ('managedbuffer', 31, 7), ('SSLProtocol', 30, 7), ('SSLObject', 30, 7), ('_SSLSocket', 40, 7), ('_SSLProtocolTransport', 30, 7), ('SelectorKey', 68, 5), ('MultiDictProxy', 20, 5), ('MultiDict', 20, 5), ('StreamReader', 17, 5), ('ResponseHandler', 11, 4), ('HttpResponseParser', 9, 4), ('TimerContext', 9, 4), ('HttpRequestParser', 20, 4), ('TaskStepMethWrapper', 4, 4), ('_FilterableJob', 272, 3), ('MimeType', 18, 3), ('AuthPhase', 9, 3), ('Timeout', 22, 2), ('URL', 42, 2), ('SplitResult', 162, 2), ('RawRequestMessage', 22, 2), ('_GeneratorContextManager', 11, 2), ('generator', 12, 2), ('HassJob', 4825, 2), ('SimpleCookie', 32, 2), ('LogEntry', 36, 2), ('WebSocketReader', 13, 2), ('CIMultiDictProxy', 30, 2), ('StreamWriter', 28, 2), ('CancelOnDisconnectRequestHandler', 21, 2), ('HomeAssistantAccessLogger', 21, 2), ('HomeAssistantRequest', 21, 2), ('UrlMappingMatchInfo', 21, 2), ('suppress', 8, 2), ('slice', 49, 1), ('hamt', 159, 1), ('ConnectionKey', 6, 1), ('tuple_iterator', 2, 1), ('TemplateStateFromEntityId', 81, 1), ('Response', 6, 1), ('PayloadAccessError', 3, 1), ('FileResponse', 7, 1), ('ClientConnectionError', 1, 1)]

The profiler seems to make my memory usage more variable, but it definitely isn't increasing.

I just now restarted HA and started the profiler to test my theory.

What could it mean if the profiler service is cleaning up the memory usage?

side note: restarting HA caused the homeassistant.components.websocket_api.http.connection errors while HA was finishing the startup process:

2023-06-17 15:47:04.664 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140224294866000] justinw from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":13481,"type":"result","success":true,"result":null}
2023-06-17 15:47:07.268 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140224329514256] justinw from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4110,"type":"event","event":{"type":"added","notifications":{"invalid_config":{"message":"The following integrations and platforms could not be set up:\n\n - browser_mod ([Show logs](/config/logs?filter=browser_mod))\n\nPlease check your config and [logs](/config/logs).","notification_id":"invalid_config","title":"Invalid config","created_at":"2023-06-17T21:47:06.611538+00:00"}}}}
2023-06-17 15:47:14.349 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140224017995024] justinw from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":4484,"type":"result","success":true,"result":null}
2023-06-17 15:47:18.464 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140224246913744] justinw from 192.168.1.1 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":6563,"type":"result","success":true,"result":null}
2023-06-17 15:47:18.696 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140224029248272] justinw from 192.168.1.184 (Mozilla/5.0 (Linux; Android 10; LenovoCD-24502F Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.186 Mobile Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: {"id":5098,"type":"result","success":true,"result":null}
bdraco commented 1 year ago

I think its happening in ha-menu-button.ts when its in narrow mode

bdraco commented 1 year ago

Which is strange since it hasn't changed in a long time

bdraco commented 1 year ago

I can get it to do it only in narrow mode

Screenshot 2023-06-17 at 6 05 36 PM
bdraco commented 1 year ago

confirmed thats where its coming from

bdraco commented 1 year ago
Screenshot 2023-06-17 at 6 13 46 PM
backcountrymountains commented 1 year ago

Where is ha-menu-button used in the frontend and how do I kill it with fire?

bdraco commented 1 year ago

Bdraco had a look in my (profiler) logs and he said there are no obvious task leaks in the them, so I'm not sure if it is an issue in core or frontend.

@VinceRMP Your initial suspicions were correct and frontend turned out to be the winner here.

ondras12345 commented 1 year ago

Why doesn't core release the memory it had allocated after the persistent_notification/subscribe messages stop coming? I have tested this by disabling Wi-Fi on the mobile phone I use for 24/7 dashboard display. The 1GB extra memory usage never decreased.

bdraco commented 1 year ago

Why doesn't core release the memory it had allocated after the persistent_notification/subscribe messages stop coming? I have tested this by disabling Wi-Fi on the mobile phone I use for 24/7 dashboard display. The 1GB extra memory usage never decreased.

I think it’s because we suppress the cancellation https://github.com/home-assistant/core/pull/94780

ondras12345 commented 1 year ago

With the patch from https://github.com/home-assistant/core/pull/94780 applied, I let it leak about 400MB and then killed the misbehaving client. Memory usage hasn't decreased yet: image I'll leave it like this overnight and report if it does release the extra memory.

Edit: I applied the patch yesterday, version https://github.com/home-assistant/core/pull/94780/commits/12d6846a39b6a48f6cbbe6738f46b1678082cb3e

bdraco commented 1 year ago

@ondras12345 Please continue in https://github.com/home-assistant/core/pull/94780 (trying the latest commit) since this issue was for the frontend and that problem has already been fixed.