home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
4.57k stars 940 forks source link

Memory leak (outside home-assistant?) #3312

Closed Werewolfke closed 2 months ago

Werewolfke commented 3 months ago

Describe the issue you are experiencing

The vm hassio has 6Gb of ram, and was using almost all of it. Looking in homeassistant itself, it says it was using only 0.8Gb, rebooting homeassistant didn't help, just a little bit, a small dip, yet it was still above 5Gb of usage

I had to do a full reboot of the virtual machine for it to clear. Uptime of the full vm is around a week, home assistant itself probably has rebooted since then, since it updates automatically and reboots after.

Maybe something in the host-logs show something? (I rebooted around 21h, so weird that nothing shows between 19h20 and then... Or even after the reboot... Screenshot shows it gradually climbing... Since the boot of the vm.

Core 2024.4.3 Supervisor 2024.04.1 Operating System 12.2 Frontend 20240404.2

What operating system image do you use?

generic-x86-64 (Generic UEFI capable x86-64 systems)

What version of Home Assistant Operating System is installed?

12.2

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

  1. Boot hassio
  2. Use it for a week
  3. Poof ...

Anything in the Supervisor logs that might be useful for us?

Not that I see or saw...

Anything in the Host logs that might be useful for us?

2024-04-15 19:07:28.515 homeassistant kernel: veth6e99efc: entered promiscuous mode
2024-04-15 19:07:28.515 homeassistant kernel: kauditd_printk_skb: 318 callbacks suppressed
2024-04-15 19:07:28.515 homeassistant kernel: audit: type=1700 audit(1713208048.514:122): dev=veth6e99efc prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
2024-04-15 19:07:28.516 homeassistant kernel: audit: type=1300 audit(1713208048.514:122): arch=c000003e syscall=44 success=yes exit=40 a0=c a1=c0013fd590 a2=28 a3=0 items=0 ppid=1 pid=498 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=unconfined key=(null)
2024-04-15 19:07:28.516 homeassistant kernel: audit: type=1327 audit(1713208048.514:122): proctitle=2F7573722F62696E2F646F636B657264002D480066643A2F2F002D2D636F6E7461696E6572643D2F72756E2F636F6E7461696E6572642F636F6E7461696E6572642E736F636B
2024-04-15 19:07:28.516 homeassistant NetworkManager[414]: <info>  [1713208048.5150] manager: (veth8bf85bb): new Veth device (/org/freedesktop/NetworkManager/Devices/11)
2024-04-15 19:07:28.516 homeassistant NetworkManager[414]: <info>  [1713208048.5153] manager: (veth6e99efc): new Veth device (/org/freedesktop/NetworkManager/Devices/12)
2024-04-15 19:07:28.570 homeassistant systemd[1]: Started libcontainer container 90373eb8f151e4002d9c15f877d9ec5a87b0d5415581e2a49f7dc65390fc2747.
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1334 audit(1713208048.573:123): prog-id=26 op=LOAD
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1334 audit(1713208048.573:124): prog-id=27 op=LOAD
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1300 audit(1713208048.573:124): arch=c000003e syscall=321 success=yes exit=16 a0=5 a1=c00016d928 a2=78 a3=0 items=0 ppid=1115 pid=1125 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=unconfined key=(null)
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1327 audit(1713208048.573:124): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F39303337336562386631353165343030326439633135663837
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1334 audit(1713208048.573:125): prog-id=28 op=LOAD
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1300 audit(1713208048.573:125): arch=c000003e syscall=321 success=yes exit=18 a0=5 a1=c00016d6c0 a2=78 a3=0 items=0 ppid=1115 pid=1125 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=unconfined key=(null)
2024-04-15 19:07:28.575 homeassistant kernel: audit: type=1327 audit(1713208048.573:125): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F39303337336562386631353165343030326439633135663837
2024-04-15 19:07:28.614 homeassistant kernel: eth0: renamed from veth8bf85bb
2024-04-15 19:07:28.620 homeassistant NetworkManager[414]: <info>  [1713208048.6201] device (veth6e99efc): carrier: link connected
2024-04-15 19:07:28.620 homeassistant kernel: hassio: port 3(veth6e99efc) entered blocking state
2024-04-15 19:07:28.620 homeassistant kernel: hassio: port 3(veth6e99efc) entered forwarding state
2024-04-15 19:07:28.724 homeassistant NetworkManager[414]: <info>  [1713208048.7236] manager: (veth051cb41): new Veth device (/org/freedesktop/NetworkManager/Devices/13)
2024-04-15 19:07:28.724 homeassistant NetworkManager[414]: <info>  [1713208048.7240] manager: (veth77c93b3): new Veth device (/org/freedesktop/NetworkManager/Devices/14)
2024-04-15 19:07:28.724 homeassistant kernel: hassio: port 4(veth77c93b3) entered blocking state
2024-04-15 19:07:28.724 homeassistant kernel: hassio: port 4(veth77c93b3) entered disabled state
2024-04-15 19:07:28.724 homeassistant kernel: veth77c93b3: entered allmulticast mode
2024-04-15 19:07:28.724 homeassistant kernel: veth77c93b3: entered promiscuous mode
2024-04-15 19:07:28.724 homeassistant kernel: hassio: port 4(veth77c93b3) entered blocking state
2024-04-15 19:07:28.724 homeassistant kernel: hassio: port 4(veth77c93b3) entered forwarding state
2024-04-15 19:07:28.775 homeassistant systemd[1]: Started libcontainer container 906c3342cc962857f36c0f6501d2141fa07e0f5aaf89c1890c4e6180814ec9fc.
2024-04-15 19:07:28.813 homeassistant kernel: eth0: renamed from veth051cb41
2024-04-15 19:07:28.828 homeassistant NetworkManager[414]: <info>  [1713208048.8287] device (veth77c93b3): carrier: link connected
2024-04-15 19:07:28.943 homeassistant kernel: hassio: port 5(veth6ae3047) entered blocking state
2024-04-15 19:07:28.943 homeassistant kernel: hassio: port 5(veth6ae3047) entered disabled state
2024-04-15 19:07:28.943 homeassistant kernel: veth6ae3047: entered allmulticast mode
2024-04-15 19:07:28.943 homeassistant kernel: veth6ae3047: entered promiscuous mode
2024-04-15 19:07:28.943 homeassistant kernel: hassio: port 5(veth6ae3047) entered blocking state
2024-04-15 19:07:28.943 homeassistant kernel: hassio: port 5(veth6ae3047) entered forwarding state
2024-04-15 19:07:28.943 homeassistant NetworkManager[414]: <info>  [1713208048.9437] manager: (veth793f19f): new Veth device (/org/freedesktop/NetworkManager/Devices/15)
2024-04-15 19:07:28.944 homeassistant NetworkManager[414]: <info>  [1713208048.9441] manager: (veth6ae3047): new Veth device (/org/freedesktop/NetworkManager/Devices/16)
2024-04-15 19:07:28.991 homeassistant systemd[1]: Started libcontainer container c220fd9b6586a5aa4cc476c9fe295d835213d92fb63b099bc2496b707b71dc37.
2024-04-15 19:07:29.036 homeassistant kernel: eth0: renamed from veth793f19f
2024-04-15 19:07:29.045 homeassistant NetworkManager[414]: <info>  [1713208049.0452] device (veth6ae3047): carrier: link connected
2024-04-15 19:07:29.174 homeassistant systemd[1]: Started libcontainer container 46dc738ec9f7e2cc00b5bd5475ab9172752bef987c4250d4d825c93c35462f70.
2024-04-15 19:07:29.476 homeassistant systemd[1]: Bluetooth service was skipped because of an unmet condition check (ConditionPathIsDirectory=/sys/class/bluetooth).
2024-04-15 19:07:29.838 homeassistant systemd[1]: Started libcontainer container 0193cfa3c2ddbaae94f36212fd8db62efef7db081211de8358c8f39ff9542739.
2024-04-15 19:07:31.771 homeassistant systemd[1]: Bluetooth service was skipped because of an unmet condition check (ConditionPathIsDirectory=/sys/class/bluetooth).
2024-04-15 19:07:31.777 homeassistant kernel: Bluetooth: Core ver 2.22
2024-04-15 19:07:31.777 homeassistant kernel: NET: Registered PF_BLUETOOTH protocol family
2024-04-15 19:07:31.777 homeassistant kernel: Bluetooth: HCI device and connection manager initialized
2024-04-15 19:07:31.777 homeassistant kernel: Bluetooth: HCI socket layer initialized
2024-04-15 19:07:31.777 homeassistant kernel: Bluetooth: L2CAP socket layer initialized
2024-04-15 19:07:31.777 homeassistant kernel: Bluetooth: SCO socket layer initialized
2024-04-15 19:07:38.397 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
2024-04-15 19:07:55.833 homeassistant systemd[1]: systemd-hostnamed.service: Deactivated successfully.
2024-04-15 19:07:55.865 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
2024-04-15 19:07:55.870 homeassistant kernel: kauditd_printk_skb: 150 callbacks suppressed
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.868:178): prog-id=25 op=UNLOAD
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.868:179): prog-id=24 op=UNLOAD
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.868:180): prog-id=23 op=UNLOAD
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.869:181): prog-id=14 op=UNLOAD
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.869:182): prog-id=13 op=UNLOAD
2024-04-15 19:07:55.870 homeassistant kernel: audit: type=1334 audit(1713208075.869:183): prog-id=12 op=UNLOAD
2024-04-15 19:10:35.139 homeassistant systemd[1]: var-lib-docker-overlay2-2942b7c53ea0e2e906122c722227b03f9ede99ed97031f1f8c2e9f26a28f6016\x2dinit-merged.mount: Deactivated successfully.
2024-04-15 19:10:35.140 homeassistant systemd[1]: mnt-data-docker-overlay2-2942b7c53ea0e2e906122c722227b03f9ede99ed97031f1f8c2e9f26a28f6016\x2dinit-merged.mount: Deactivated successfully.
2024-04-15 19:10:35.220 homeassistant systemd[1]: Started libcontainer container 1c0fc12820ea5446d74c44eec019985804192f21949eaf452f4dd81d93b249bf.
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1334 audit(1713208235.222:184): prog-id=45 op=LOAD
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1334 audit(1713208235.222:185): prog-id=46 op=LOAD
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1300 audit(1713208235.222:185): arch=c000003e syscall=321 success=yes exit=16 a0=5 a1=c00019d928 a2=78 a3=0 items=0 ppid=2188 pid=2198 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=unconfined key=(null)
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1327 audit(1713208235.222:185): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F31633066633132383230656135343436643734633434656563
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1334 audit(1713208235.222:186): prog-id=47 op=LOAD
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1300 audit(1713208235.222:186): arch=c000003e syscall=321 success=yes exit=18 a0=5 a1=c00019d6c0 a2=78 a3=0 items=0 ppid=2188 pid=2198 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=unconfined key=(null)
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1327 audit(1713208235.222:186): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F31633066633132383230656135343436643734633434656563
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1334 audit(1713208235.222:187): prog-id=47 op=UNLOAD
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1300 audit(1713208235.222:187): arch=c000003e syscall=3 success=yes exit=0 a0=12 a1=0 a2=0 a3=0 items=0 ppid=2188 pid=2198 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc" exe="/usr/bin/runc" subj=unconfined key=(null)
2024-04-15 19:10:35.224 homeassistant kernel: audit: type=1327 audit(1713208235.222:187): proctitle=72756E63002D2D726F6F74002F7661722F72756E2F646F636B65722F72756E74696D652D72756E632F6D6F6279002D2D6C6F67002F72756E2F636F6E7461696E6572642F696F2E636F6E7461696E6572642E72756E74696D652E76322E7461736B2F6D6F62792F31633066633132383230656135343436643734633434656563
2024-04-15 19:10:40.313 homeassistant kernel: kauditd_printk_skb: 6 callbacks suppressed
2024-04-15 19:10:40.313 homeassistant kernel: audit: type=1334 audit(1713208240.311:190): prog-id=49 op=LOAD
2024-04-15 19:10:40.313 homeassistant kernel: audit: type=1334 audit(1713208240.311:191): prog-id=50 op=LOAD
2024-04-15 19:10:40.313 homeassistant kernel: audit: type=1334 audit(1713208240.311:192): prog-id=51 op=LOAD
2024-04-15 19:10:40.319 homeassistant systemd[1]: Starting Hostname Service...
2024-04-15 19:10:40.361 homeassistant systemd[1]: Started Hostname Service.
2024-04-15 19:10:40.365 homeassistant kernel: audit: type=1334 audit(1713208240.363:193): prog-id=52 op=LOAD
2024-04-15 19:10:40.365 homeassistant kernel: audit: type=1334 audit(1713208240.363:194): prog-id=53 op=LOAD
2024-04-15 19:10:40.365 homeassistant kernel: audit: type=1334 audit(1713208240.363:195): prog-id=54 op=LOAD
2024-04-15 19:10:40.374 homeassistant systemd[1]: Starting Time & Date Service...
2024-04-15 19:10:40.411 homeassistant systemd[1]: Started Time & Date Service.
2024-04-15 19:10:44.312 homeassistant dockerd[498]: time="2024-04-15T19:10:44.312314884Z" level=error msg="[resolver] failed to query DNS server: 192.168.1.250:53, query: ;version.home-assistant.io.\tIN\t A" error="read udp 172.30.232.2:39256->192.168.1.250:53: i/o timeout"
2024-04-15 19:11:10.381 homeassistant systemd[1]: systemd-hostnamed.service: Deactivated successfully.
2024-04-15 19:11:10.415 homeassistant kernel: audit: type=1334 audit(1713208270.412:196): prog-id=51 op=UNLOAD
2024-04-15 19:11:10.415 homeassistant kernel: audit: type=1334 audit(1713208270.412:197): prog-id=50 op=UNLOAD
2024-04-15 19:11:10.415 homeassistant kernel: audit: type=1334 audit(1713208270.412:198): prog-id=49 op=UNLOAD
2024-04-15 19:11:10.443 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
2024-04-15 19:11:10.451 homeassistant kernel: audit: type=1334 audit(1713208270.449:199): prog-id=54 op=UNLOAD
2024-04-15 19:11:10.451 homeassistant kernel: audit: type=1334 audit(1713208270.449:200): prog-id=53 op=UNLOAD
2024-04-15 19:11:10.451 homeassistant kernel: audit: type=1334 audit(1713208270.449:201): prog-id=52 op=UNLOAD
2024-04-15 19:19:49.015 homeassistant kernel: audit: type=1334 audit(1713208789.013:202): prog-id=55 op=LOAD
2024-04-15 19:19:49.015 homeassistant systemd-timesyncd[481]: Network configuration changed, trying to establish connection.
2024-04-15 19:19:49.023 homeassistant systemd-timesyncd[481]: Contacted time server 162.159.200.1:123 (time.cloudflare.com).
2024-04-15 19:19:49.023 homeassistant systemd[1]: Started Journal Gateway Service.
2024-04-15 19:22:35.045 homeassistant systemd[1]: Starting Cleanup of Temporary Directories...
2024-04-15 19:22:35.051 homeassistant systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
2024-04-15 19:22:35.052 homeassistant systemd[1]: Finished Cleanup of Temporary Directories.

System information

System Information

version core-2024.4.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.2
os_name Linux
os_version 6.6.25-haos
arch x86_64
timezone Europe/Brussels
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4664 Installed Version | 1.34.0 Stage | running Available Repositories | 1402 Downloaded Repositories | 30
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 12.2 -- | -- update_channel | beta supervisor_version | supervisor-2024.04.1 agent_version | 1.6.0 docker_version | 25.0.5 disk_total | 30.8 GB disk_used | 13.2 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons | Advanced SSH & Web Terminal (17.2.0), Studio Code Server (5.15.0), Music Assistant BETA (2.0.0b136), Firefox (1.1.0)
Dashboards dashboards | 9 -- | -- resources | 15 views | 16 mode | storage
Recorder oldest_recorder_run | April 5, 2024 at 17:02 -- | -- current_recorder_run | April 15, 2024 at 21:07 estimated_db_size | 371.70 MiB database_engine | sqlite database_version | 3.44.2

Additional information

Screenshot_20240415_211000 Screenshot_20240415_211024_Home Assistant Screenshot_20240415_213125_Home Assistant

Impact123 commented 3 months ago

PVE also includes cache and buffers in this stat. At least on linux VMs. See here: https://www.google.com/search?q=wrong+memory+usage+site%3Aproxmox.com If you want to show PVE statistics please show them in the actual GUI, not a random app.

Werewolfke commented 2 months ago

The random app is a proxmox monitoring app. I could not get into pve at that time. I will look into it and see what I can do, but I know I had to increase ram from 4Gb to 6 because it crashed otherwise, so I didn't think it were to be caching. If it is 100% caching then this issue can be closed.

sairon commented 2 months ago

You can check the amount of available memory directly in Home Assistant Settings -> System -> Hardware. If the system crashes, check Host logs (or Core logs) then but if it were an OOM event it would have been most likely correlated with ~100% memory usage in Proxmox statistics, which I don't see in the graphs you provided. For comparison here's hourly average of my Proxmox VM running a "production" deployment of HAOS (where the available memory in the VM stays around ~2GB still):

image

All in all I don't see anything wrong here.

Impact123 commented 2 months ago

You can check the used/free memory in more detail with the glances addon or by running free -h via SSH. I recommend you visit the discord server if you need help troubleshooting.