home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
5.01k stars 981 forks source link

Home Assistant Crashes when ESPHome runs out of memory (used to just crash ESPHome) #3184

Open roblamoreaux opened 8 months ago

roblamoreaux commented 8 months ago

Describe the issue you are experiencing

When I compile NS-Panel Blueprint on my Odroid N2+ (4GB) with too many other things running it used to restart the ESPHome addon. Now with ESPHome 2024.2.0beta (and 2023.12.2) and OS version 12.0.rc1, supervisor 2024.02.0, and Corte 2024.2.2, it causes Home Assistant to restart. Turning off other add-ons such as Music Assistant reduces the chance of crash since ESPHome can get more resources.

What operating system image do you use?

odroid-n2 (Hardkernel ODROID-N2/N2+)

What version of Home Assistant Operating System is installed?

12.0.rc1

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

  1. Compile NS-Panel Blueprint esphome code with more than 80% memory free on Odroid N2+ which used to cause ESPHome to restart.
  2. It gets most of the way through and then HA restarts and supervisor log is restarted leaving no evidence of pre-crash info
  3. Note I already reduced process limit to 1 on ESPHome. ... I am not use if this is ESPHome, Core or OS, but it used to just kill ESPHome which I could deal with

Anything in the Supervisor logs that might be useful for us?

24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/ad4c456f/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/d557e82e/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/a6ab9a92/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/4ded7296/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/49cb62d8/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/13d28fa7/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/d7a74702/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/7c6163a7/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/ecafd656/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/833ebb4e/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/a68d9e51/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/19ec4a64/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/dbb5f422/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/8998f60f/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/68fada8f/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/610b6b9e/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/f8b4fa7a/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/c8ccbc97/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/80f54172/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/c5de3154/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/50e40a8b/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/0f7903f9/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/730f3cd2/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/e57555ea/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/c2f90a79/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/3958110a/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/504f88eb/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/6794c7bb/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/8ac1a916/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/c5b54025/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/65c7614d/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/fefd3657/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/087bdfb9/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/4bdfc79d/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/5f86353b/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/4c6fe0d5/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/c068652c/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/4e3eec7f/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/562fd439/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/b58b769b/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/690209a6/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/50441d45/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/5620eb4e/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/e303f137/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/af1b164c/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/46e42d0f/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/0b588555/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/1632964e/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:08 INFO (MainThread) [supervisor.api.middleware.security] /backups/b5453547/info access from cebe7a76_hassio_google_drive_backup
24-02-19 09:28:09 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-almond with version 3.0.0
24-02-19 09:28:09 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:09 INFO (SyncWorker_5) [supervisor.docker.manager] Cleaning addon_d5369777_music_assistant_beta application
24-02-19 09:28:09 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:10 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/music-assistant/server with version 2.0.0b100
24-02-19 09:28:10 INFO (SyncWorker_2) [supervisor.docker.manager] Cleaning addon_db21ed7f_inadyn application
24-02-19 09:28:10 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:12 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/alexbelgium/inadyn-aarch64 with version 2.12.0-2
24-02-19 09:28:12 INFO (SyncWorker_0) [supervisor.docker.manager] Cleaning addon_a0d7b954_wireguard application
24-02-19 09:28:12 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:13 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:14 WARNING (MainThread) [supervisor.api.ingress] No valid ingress session 2188ba436cd8ccf7847bcad4c7e280267cd24a1001777c7e2fdcad22613de82e1b3802cfc32f9ff00117d222621960d8a1d4174261174878bdd31b3c7fcd8776
24-02-19 09:28:15 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/wireguard/aarch64 with version 0.10.2
24-02-19 09:28:17 INFO (MainThread) [supervisor.api.middleware.security] /dns/info access from a0d7b954_wireguard
24-02-19 09:28:17 WARNING (MainThread) [supervisor.api.middleware.security] No API token provided for /core
24-02-19 09:28:39 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled
24-02-19 09:28:39 INFO (MainThread) [supervisor.core] Supervisor is up and running
24-02-19 09:28:39 INFO (MainThread) [supervisor.host.info] Updating local host information
24-02-19 09:28:39 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/beta.json
24-02-19 09:28:39 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
24-02-19 09:28:39 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
24-02-19 09:28:39 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
24-02-19 09:28:39 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'interface' does not exist in the schema for Samba share (core_samba)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'anonymous' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'connection' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'address' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'try_private' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'start_type' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'topic' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'notifications' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'clientid' does not exist in the schema for Mosquitto broker (core_mosquitto)
24-02-19 09:28:40 WARNING (MainThread) [supervisor.addons.options] Option 'mqtt_use_legacy_entities' does not exist in the schema for Samba NAS (1a32f091_sambanas)
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.check] System checks complete
24-02-19 09:28:40 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
24-02-19 09:28:41 INFO (MainThread) [supervisor.host.services] Updating service information
24-02-19 09:28:41 INFO (MainThread) [supervisor.host.network] Updating local network information
24-02-19 09:28:41 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
24-02-19 09:28:41 INFO (MainThread) [supervisor.host.manager] Host information reload completed
24-02-19 09:28:41 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
24-02-19 09:28:41 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
24-02-19 09:28:41 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete

Anything in the Host logs that might be useful for us?

Feb 19 14:28:41 homeassistant kernel: audit: type=1701 audit(1708352921.238:525): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=8953 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.262:526): prog-id=136 op=LOAD
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.266:527): prog-id=137 op=LOAD
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.266:528): prog-id=138 op=LOAD
Feb 19 14:28:41 homeassistant systemd[1]: Started Process Core Dump (PID 8956/UID 0).
Feb 19 14:28:41 homeassistant systemd-coredump[8958]: Process 8953 (genie-client) of user 0 dumped core.
Feb 19 14:28:41 homeassistant systemd[1]: systemd-coredump@3-8956-0.service: Deactivated successfully.
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.902:529): prog-id=138 op=UNLOAD
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.902:530): prog-id=137 op=UNLOAD
Feb 19 14:28:41 homeassistant kernel: audit: type=1334 audit(1708352921.902:531): prog-id=136 op=UNLOAD
Feb 19 14:28:46 homeassistant kernel: audit: type=1701 audit(1708352926.846:532): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=8986 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:28:46 homeassistant kernel: audit: type=1334 audit(1708352926.862:533): prog-id=139 op=LOAD
Feb 19 14:28:46 homeassistant kernel: audit: type=1334 audit(1708352926.862:534): prog-id=140 op=LOAD
Feb 19 14:28:46 homeassistant kernel: audit: type=1334 audit(1708352926.862:535): prog-id=141 op=LOAD
Feb 19 14:28:46 homeassistant systemd[1]: Started Process Core Dump (PID 8989/UID 0).
Feb 19 14:28:47 homeassistant systemd-coredump[8990]: Process 8986 (genie-client) of user 0 dumped core.
Feb 19 14:28:47 homeassistant systemd[1]: systemd-coredump@4-8989-0.service: Deactivated successfully.
Feb 19 14:28:47 homeassistant kernel: audit: type=1334 audit(1708352927.382:536): prog-id=141 op=UNLOAD
Feb 19 14:28:47 homeassistant kernel: audit: type=1334 audit(1708352927.382:537): prog-id=140 op=UNLOAD
Feb 19 14:28:47 homeassistant kernel: audit: type=1334 audit(1708352927.382:538): prog-id=139 op=UNLOAD
Feb 19 14:28:52 homeassistant kernel: audit: type=1701 audit(1708352932.370:539): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9014 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.386:540): prog-id=142 op=LOAD
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.386:541): prog-id=143 op=LOAD
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.386:542): prog-id=144 op=LOAD
Feb 19 14:28:52 homeassistant systemd[1]: Started Process Core Dump (PID 9017/UID 0).
Feb 19 14:28:52 homeassistant systemd-coredump[9018]: Process 9014 (genie-client) of user 0 dumped core.
Feb 19 14:28:52 homeassistant systemd[1]: systemd-coredump@5-9017-0.service: Deactivated successfully.
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.902:543): prog-id=144 op=UNLOAD
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.902:544): prog-id=143 op=UNLOAD
Feb 19 14:28:52 homeassistant kernel: audit: type=1334 audit(1708352932.902:545): prog-id=142 op=UNLOAD
Feb 19 14:28:57 homeassistant kernel: audit: type=1701 audit(1708352937.898:546): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9052 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:28:57 homeassistant kernel: audit: type=1334 audit(1708352937.922:547): prog-id=145 op=LOAD
Feb 19 14:28:57 homeassistant kernel: audit: type=1334 audit(1708352937.922:548): prog-id=146 op=LOAD
Feb 19 14:28:57 homeassistant kernel: audit: type=1334 audit(1708352937.922:549): prog-id=147 op=LOAD
Feb 19 14:28:57 homeassistant systemd[1]: Started Process Core Dump (PID 9055/UID 0).
Feb 19 14:28:58 homeassistant systemd-coredump[9056]: Process 9052 (genie-client) of user 0 dumped core.
Feb 19 14:28:58 homeassistant systemd[1]: systemd-coredump@6-9055-0.service: Deactivated successfully.
Feb 19 14:28:58 homeassistant kernel: audit: type=1334 audit(1708352938.534:550): prog-id=147 op=UNLOAD
Feb 19 14:28:58 homeassistant kernel: audit: type=1334 audit(1708352938.534:551): prog-id=146 op=UNLOAD
Feb 19 14:28:58 homeassistant kernel: audit: type=1334 audit(1708352938.534:552): prog-id=145 op=UNLOAD
Feb 19 14:29:03 homeassistant kernel: audit: type=1701 audit(1708352943.502:553): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9141 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:29:03 homeassistant kernel: audit: type=1334 audit(1708352943.522:554): prog-id=148 op=LOAD
Feb 19 14:29:03 homeassistant kernel: audit: type=1334 audit(1708352943.522:555): prog-id=149 op=LOAD
Feb 19 14:29:03 homeassistant kernel: audit: type=1334 audit(1708352943.522:556): prog-id=150 op=LOAD
Feb 19 14:29:03 homeassistant systemd[1]: Started Process Core Dump (PID 9146/UID 0).
Feb 19 14:29:04 homeassistant systemd-coredump[9147]: Process 9141 (genie-client) of user 0 dumped core.
Feb 19 14:29:04 homeassistant systemd[1]: systemd-coredump@7-9146-0.service: Deactivated successfully.
Feb 19 14:29:04 homeassistant kernel: audit: type=1334 audit(1708352944.142:557): prog-id=150 op=UNLOAD
Feb 19 14:29:04 homeassistant kernel: audit: type=1334 audit(1708352944.142:558): prog-id=149 op=UNLOAD
Feb 19 14:29:04 homeassistant kernel: audit: type=1334 audit(1708352944.142:559): prog-id=148 op=UNLOAD
Feb 19 14:29:09 homeassistant kernel: audit: type=1701 audit(1708352949.098:560): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9222 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.114:561): prog-id=151 op=LOAD
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.114:562): prog-id=152 op=LOAD
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.114:563): prog-id=153 op=LOAD
Feb 19 14:29:09 homeassistant systemd[1]: Started Process Core Dump (PID 9225/UID 0).
Feb 19 14:29:09 homeassistant systemd-coredump[9226]: Process 9222 (genie-client) of user 0 dumped core.
Feb 19 14:29:09 homeassistant systemd[1]: systemd-coredump@8-9225-0.service: Deactivated successfully.
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.734:564): prog-id=153 op=UNLOAD
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.734:565): prog-id=152 op=UNLOAD
Feb 19 14:29:09 homeassistant kernel: audit: type=1334 audit(1708352949.734:566): prog-id=151 op=UNLOAD
Feb 19 14:29:10 homeassistant systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Feb 19 14:29:10 homeassistant kernel: audit: type=1334 audit(1708352950.638:567): prog-id=132 op=UNLOAD
Feb 19 14:29:10 homeassistant kernel: audit: type=1334 audit(1708352950.638:568): prog-id=131 op=UNLOAD
Feb 19 14:29:10 homeassistant kernel: audit: type=1334 audit(1708352950.638:569): prog-id=130 op=UNLOAD
Feb 19 14:29:11 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
Feb 19 14:29:14 homeassistant kernel: kauditd_printk_skb: 3 callbacks suppressed
Feb 19 14:29:14 homeassistant kernel: audit: type=1701 audit(1708352954.690:573): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9262 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:29:14 homeassistant kernel: audit: type=1334 audit(1708352954.702:574): prog-id=154 op=LOAD
Feb 19 14:29:14 homeassistant kernel: audit: type=1334 audit(1708352954.702:575): prog-id=155 op=LOAD
Feb 19 14:29:14 homeassistant kernel: audit: type=1334 audit(1708352954.702:576): prog-id=156 op=LOAD
Feb 19 14:29:14 homeassistant systemd[1]: Started Process Core Dump (PID 9265/UID 0).
Feb 19 14:29:15 homeassistant systemd-coredump[9266]: Process 9262 (genie-client) of user 0 dumped core.
Feb 19 14:29:15 homeassistant systemd[1]: systemd-coredump@9-9265-0.service: Deactivated successfully.
Feb 19 14:29:15 homeassistant kernel: audit: type=1334 audit(1708352955.314:577): prog-id=156 op=UNLOAD
Feb 19 14:29:15 homeassistant kernel: audit: type=1334 audit(1708352955.314:578): prog-id=155 op=UNLOAD
Feb 19 14:29:15 homeassistant kernel: audit: type=1334 audit(1708352955.314:579): prog-id=154 op=UNLOAD
Feb 19 14:29:20 homeassistant kernel: audit: type=1701 audit(1708352960.251:580): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9284 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.267:581): prog-id=157 op=LOAD
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.267:582): prog-id=158 op=LOAD
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.267:583): prog-id=159 op=LOAD
Feb 19 14:29:20 homeassistant systemd[1]: Started Process Core Dump (PID 9287/UID 0).
Feb 19 14:29:20 homeassistant systemd-coredump[9288]: Process 9284 (genie-client) of user 0 dumped core.
Feb 19 14:29:20 homeassistant systemd[1]: systemd-coredump@10-9287-0.service: Deactivated successfully.
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.887:584): prog-id=159 op=UNLOAD
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.887:585): prog-id=158 op=UNLOAD
Feb 19 14:29:20 homeassistant kernel: audit: type=1334 audit(1708352960.887:586): prog-id=157 op=UNLOAD
Feb 19 14:29:25 homeassistant kernel: audit: type=1701 audit(1708352965.835:587): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=9326 comm="genie-client" exe="/usr/local/bin/genie-client" sig=5 res=1
Feb 19 14:29:25 homeassistant kernel: audit: type=1334 audit(1708352965.851:588): prog-id=160 op=LOAD
Feb 19 14:29:25 homeassistant kernel: audit: type=1334 audit(1708352965.851:589): prog-id=161 op=LOAD
Feb 19 14:29:25 homeassistant kernel: audit: type=1334 audit(1708352965.851:590): prog-id=162 op=LOAD
Feb 19 14:29:25 homeassistant systemd[1]: Started Process Core Dump (PID 9329/UID 0).
Feb 19 14:29:26 homeassistant systemd-coredump[9330]: Process 9326 (genie-client) of user 0 dumped core.
Feb 19 14:29:26 homeassistant systemd[1]: systemd-coredump@11-9329-0.service: Deactivated successfully.
Feb 19 14:29:26 homeassistant kernel: audit: type=1334 audit(1708352966.435:591): prog-id=162 op=UNLOAD
Feb 19 14:29:26 homeassistant kernel: audit: type=1334 audit(1708352966.435:592): prog-id=161 op=UNLOAD
Feb 19 14:29:26 homeassistant kernel: audit: type=1334 audit(1708352966.435:593): prog-id=160 op=UNLOAD
Feb 19 14:29:27 homeassistant kernel: audit: type=1334 audit(1708352967.119:594): prog-id=163 op=LOAD
Feb 19 14:29:27 homeassistant systemd-timesyncd[496]: Network configuration changed, trying to establish connection.
Feb 19 14:29:27 homeassistant systemd[1]: Started Journal Gateway Service.
Feb 19 14:29:27 homeassistant systemd-timesyncd[496]: Contacted time server 162.159.200.1:123 (time.cloudflare.com).
Feb 19 14:29:27 homeassistant systemd-journal-gatewayd[9341]: microhttpd: MHD_OPTION_EXTERNAL_LOGGER is not the first option specified for the daemon. Some messages may be printed by the standard MHD logger.

System information

None

Additional information

As noted above I have had an issue where compiling NS-Panel Blueprint ESPHome Yaml code would crash ESPHome due to insufficent resources and stopping other add-ons (especially Music Assistant) would free up enough resources for the compile to work. Now instead of restarting ESPHome the whole system restarts. This behavior is much worse where restarting ESPHome was acceptable if not desirable.

roblamoreaux commented 8 months ago

I mistyped above and the non-beta version of ESPHome is 2023.12.9.

XiaoXianNv-boot commented 8 months ago

Looks like there's not enough RAM

roblamoreaux commented 8 months ago

Looks like there's not enough RAM

yes I know that it is insufficient RAM. my issue is it no longer kills ESPHome but causes the whole system to restart

XiaoXianNv-boot commented 8 months ago

add swapfile

sairon commented 8 months ago

@roblamoreaux By the "whole system" you mean the OS as whole or just main system components running in containers? I expect when it runs out of memory, the OOM killer starts killing things. It may eventually kill containerd, which would probably end up with all containers being restarted. Also I'd expect some messages to show in the host logs then, but maybe they're only in the host dmesg? :thinking: Can you check with HDMI and keyboard attached?

In any case, if you're running low on resources and the compilation is very demanding, there's not much we can do on OS side to make it pass. But I absolutely agree it should try to behave well and it should be obvious what went bad.

roblamoreaux commented 8 months ago

I'll plug the monitor in tonight and try compiling again. I'm not sure how much restarts, just know that everything stops responding then the browser shows the core start up messages.

I agree well behaved is the addon getting restarted and the core HA tasks keep running. That is what I saw before this.

roblamoreaux commented 8 months ago

So this morning I updated to ESPHome 2024.2.0 hooked up the monitor and keyboard and tried again. The monitor showed the same messages it does when I power up the Odroid and at the CLI prompt I can't get any response except the same error message. Everything else seems to work, so I'll try to get a chance tonight or tomorrow to SSH as root and see if I can get anything different. Hopefully the pictures below come through. If not I'll try again after I get to work.

20240221_072624 20240221_064954

roblamoreaux commented 8 months ago

wrong second picture above... 20240221_065153

woutf commented 6 months ago

I seem to experience this same issue while trying to update my S3-BOX to 2024.4.2 The memory consumption shoots up at one point during compiling and Home Assistant then reboots. Screenshot 2024-05-02 145209

daviss57 commented 5 months ago

Same issue here. Installation onto S3 BOX stops at the same point everytime and causes HA to restart. image

image

image

cogentcoder commented 3 months ago

My home assistant is also crashing while updating or compiling yaml for ESP32 Devkit. I can update and compile for ESP8266 without issue. I am using Raspberry Pi 3B with 1GB, so it should be RAM issue and I want to update but not sure to go with NUC system or Raspberry Pi and then how much is enough ram is also not clear, people with 4GB is having a problem so how much should be good enough 8GB? 16GB? Till then, I have installed python and esphome on windows 11 and can compile esphome yaml under a minute and then update the program using esphome web.

edm00se commented 2 months ago

I've also started seeing esphome in my HA add-on take down my entire HA instance on my Raspberry Pi 4 ~(8GB)~ which had been previously more than capable. It started for me in late July. It appears I it doesn't make it far into compiling before it takes down HA and I lose my connection.

TexTown commented 2 months ago

Interesting, I thought this was memory related because my Raspberry Pi only has 2gb of memory. But if this happens on 8gb systems as well, it becomes even harder to explain.

I can still work around the issue by shutting down most addons (Grafana, smb, ssh, matter etc) and then performing the esp update. Does that work for you?

edm00se commented 2 months ago

@TexTown after stopping a couple of other add ons I was able to successfully compile and install for the update in esphome. I suspect your observation on the memory is accurate, in whatever way that HA does or doesn't manage memory for and between add ons. Also of note, I forgot which Pi was which and I'm also running on a 2GB.

dpgh947 commented 2 months ago

Install this to increase your swap space. It's an addon, but it only runs at startup to add the extra swap space, and then ends. https://github.com/TazzerMAN/increase_swap_addon

TexTown commented 2 months ago

I've installed the addon, increased the swap size to 2GB and can now successfully update ESPHome devices without the need to stop other addons. This is great, thanks @dpgh947 for the tip.

dpgh947 commented 2 months ago

Brilliant, you're welcome. I also have a 2gb pi4 and I also had trouble with esphome compiles.

nielsaust commented 2 weeks ago

Install this to increase your swap space. It's an addon, but it only runs at startup to add the extra swap space, and then ends. https://github.com/TazzerMAN/increase_swap_addon

I want to thank you greatly for this tip. My HA install rebooted not only when compiling for ESPHome but also other instances and this seemed to have solved it.

So... Thanks!

dpgh947 commented 2 weeks ago

Cheers, you're welcome