Closed rck777 closed 2 years ago
Hm, this is indeed interesting. It would be interesting which process' memory increases. Can you login onto the OS (using SSH, see https://developers.home-assistant.io/docs/operating-system/debugging/) and run the following command?
ps aux --sort -rss | head
Output memory usage
Hello, see below the log of ps. The 'python3 -m homeassistant --config /config' proces consuption of memory is high (within minutes 85.6%
ps aux --sort -rss | head USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 5020 105 85.6 3636704 3291144 ? Ssl 19:29 5:35 python3 -m homeassistant --config /config root 1518 1.6 1.2 126140 47828 ? Ssl 19:29 0:05 python3 -m supervisor root 391 11.1 1.2 2155612 46456 ? Ssl 19:29 0:37 /usr/bin/dockerd -H fd:// --storage-driver=overlay2 --log-driver=journald --data-root /mnt/data/docker root 160 8.3 0.9 304968 37448 ? Rs 19:28 0:28 /usr/lib/systemd/systemd-journald root 399 0.5 0.5 2432832 21564 ? Ssl 19:29 0:01 containerd --config /var/run/docker/containerd/containerd.toml --log-level info root 2058 0.2 0.3 743840 12912 ? Ssl 19:29 0:00 coredns -conf /etc/corefile root 5331 0.4 0.3 27320 11988 ? Ss 19:30 0:01 /usr/bin/python3 /usr/bin/hass-configurator /etc/configurator.conf root 1150 0.0 0.1 1481364 6192 ? Ssl 19:29 0:00 docker container start --attach hassio_supervisor root 3192 0.5 0.1 94180 4592 ? S<sl 19:29 0:01 pulseaudio --system -vvv --disallow-exit --exit-idle-time=-1 --disable-shm
Problem found i think ;-) After i used ps command i also check the system messages with journalctl. Since I was in debug mode anyway. There are problems with the integration dsmr (Dutch Smart Meter Requirements). See log below. There are hundreds / thousands of error lines present about this error. So, i removed the integration and the system now seems stable.
Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: File "/usr/src/homeassistant/homeassistant/components/dsmr/sensor.py", line 186, in connect_and_reconnect Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: transport, protocol = await hass.loop.create_task(reader_factory()) Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: File "/usr/local/lib/python3.8/asyncio/coroutines.py", line 124, in coro Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: res = func(*args, *kw) Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: File "/usr/local/lib/python3.8/site-packages/serial_asyncio/init.py", line 410, in create_serial_connection Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: ser = serial.serial_for_url(args, **kwargs) Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: File "/usr/local/lib/python3.8/site-packages/serial/init.py", line 88, in serial_for_url Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: instance.open() Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: File "/usr/local/lib/python3.8/site-packages/serial/serialposix.py", line 268, in open Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg)) Dec 21 20:22:04 homeassistant fb413cc9aaba[393]: serial.serialutil.SerialException: [Errno 2] could not open port /dev/serial/by-id/usb-FTDI_P1_Converter_Cable_P17UI81-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-FTDI_P1_Converter_Cable_P17UI81-if00-port0'
Uh yeah that make sense. So did the serial device name /dev/serial/by-id/usb-FTDI_P1_Converter_Cable_P17UI81-if00-port0
change?
I did not connect any USB devices. So, there was actually no device. Only in the config. It seems a solution that a non-existent device ignored in de configuration? Config check was OK.
i removed the integration, insert P1 USB cable en enable config sensor:
The serial device name has remained the same. Right now it works
Right now it works
So I guess it was the integration then which caused the high CPU/RAM usage?
The migrated instance form PI4 to ODROID N2 is very unstable at this moment. After remove and installation DSMR it goes a bit better but still crashes after about 12 hours. With debugging options are available to debugging automations?
I believe I am experiencing the same, though I originally though it was caused by the HA update rather than migration from RPi3 to N2+: https://github.com/home-assistant/core/issues/44691
few hours after startup, HA is getting real slow: automations fire up with huge delay, my zigbee buttons react very poorly; and in around 12-15 hours goes totally unresponsive: nothing works anymore, even web UI, so I have to physically reboot N2.
I might have found the cause:
ERROR (Recorder) [homeassistant.components.recorder] Error executing query: (sqlite3.DatabaseError) database disk image is malformed
removed config/home-assistant_v2.db and CPU+memory usage is way better now. I will monitor if the freeze issue is gone completely.
I also regularly have a corrupt database file. The problem remains even after deletion and restart. I am interested in your findings
3.5 days so far, everything looking good. all I did was removing the corrupted database.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
Hardware
Home Assistant OS release:
Migration to ODROID-N2 4Gig. I make a snapshot on Raspberry PI4 and restore snapshot on the new ODROID-N2. I restored two instances. Both have the same problem
ha info
20-12-21 21:16:07 INFO (SyncWorker_0) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-ssh with version 8.9.1 20-12-21 21:16:12 INFO (MainThread) [supervisor.core] Start Home Assistant Core 20-12-21 21:16:12 INFO (SyncWorker_4) [supervisor.docker.interface] Starting homeassistant 20-12-21 21:16:12 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready 20-12-21 21:20:37 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token 20-12-21 21:21:08 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance 20-12-21 21:21:08 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.APPLICATION' starting 1 add-ons 20-12-21 21:21:08 INFO (SyncWorker_4) [supervisor.docker.interface] Cleaning addon_core_configurator application 20-12-21 21:21:09 INFO (SyncWorker_4) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-configurator with version 5.2.0 20-12-21 21:21:14 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled 20-12-21 21:21:14 INFO (MainThread) [supervisor.misc.hwmon] Started Supervisor hardware monitor 20-12-21 21:21:14 INFO (MainThread) [supervisor.core] Supervisor is up and running 20-12-21 21:21:14 INFO (MainThread) [supervisor.host.info] Updating local host information 20-12-21 21:21:14 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json 20-12-21 21:21:14 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING 20-12-21 21:21:14 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete 20-12-21 21:21:14 INFO (MainThread) [supervisor.host.services] Updating service information 20-12-21 21:21:14 INFO (MainThread) [supervisor.host.network] Updating local network information [s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] done. [cont-init.d] executing container initialization scripts... [cont-init.d] udev.sh: executing... [21:23:57] INFO: Update udev information [cont-init.d] udev.sh: exited 0. [cont-init.d] done. [services.d] starting services [services.d] done. [21:23:57] INFO: Starting local supervisor watchdog... 20-12-21 21:23:59 INFO (MainThread) [main] Initializing Supervisor setup 20-12-21 21:23:59 INFO (MainThread) [supervisor.bootstrap] Initializing Supervisor Sentry 20-12-21 21:23:59 INFO (MainThread) [supervisor.bootstrap] Seting up coresys for machine: odroid-n2 20-12-21 21:23:59 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attaching to Supervisor homeassistant/aarch64-hassio-supervisor with version 2020.12.7 20-12-21 21:23:59 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.INITIALIZE 20-12-21 21:23:59 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 20-12-21 21:23:59 INFO (MainThread) [main] Setting up Supervisor 20-12-21 21:23:59 INFO (MainThread) [supervisor.api] Starting API on 172.30.32.2 20-12-21 21:23:59 INFO (MainThread) [supervisor.host.info] Updating local host information 20-12-21 21:23:59 INFO (MainThread) [supervisor.host.services] Updating service information 20-12-21 21:24:00 INFO (MainThread) [supervisor.host.network] Updating local network information 20-12-21 21:24:00 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information 20-12-21 21:24:00 INFO (MainThread) [supervisor.host] Host information reload completed 20-12-21 21:24:00 INFO (MainThread) [supervisor.host.apparmor] Loading AppArmor Profiles: {'hassio-supervisor'} 20-12-21 21:24:00 INFO (MainThread) [supervisor.host.services] Reloading local service hassos-apparmor.service 20-12-21 21:24:00 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-hassio-dns with version 2020.11.0 20-12-21 21:24:00 INFO (MainThread) [supervisor.plugins.dns] Starting CoreDNS plugin 20-12-21 21:24:00 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_dns application 20-12-21 21:24:01 INFO (SyncWorker_0) [supervisor.docker.dns] Starting DNS homeassistant/aarch64-hassio-dns with version 2020.11.0 - 172.30.32.3 20-12-21 21:24:01 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf 20-12-21 21:24:01 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-hassio-audio with version 17 20-12-21 21:24:01 INFO (MainThread) [supervisor.plugins.audio] Starting Audio plugin 20-12-21 21:24:01 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_audio application 20-12-21 21:24:02 INFO (SyncWorker_0) [supervisor.docker.audio] Starting Audio homeassistant/aarch64-hassio-audio with version 17 - 172.30.32.4 20-12-21 21:24:02 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-hassio-cli with version 2020.11.1 20-12-21 21:24:02 INFO (MainThread) [supervisor.plugins.cli] Starting CLI plugin 20-12-21 21:24:02 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_cli application 20-12-21 21:24:02 INFO (SyncWorker_0) [supervisor.docker.cli] Starting CLI homeassistant/aarch64-hassio-cli with version 2020.11.1 - 172.30.32.5 20-12-21 21:24:02 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-hassio-observer with version 2020.10.1 20-12-21 21:24:03 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-hassio-multicast with version 3 20-12-21 21:24:03 INFO (MainThread) [supervisor.plugins.multicast] Starting Multicast plugin 20-12-21 21:24:03 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_multicast application 20-12-21 21:24:03 INFO (SyncWorker_0) [supervisor.docker.multicast] Starting Multicast homeassistant/aarch64-hassio-multicast with version 3 - Host 20-12-21 21:24:03 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json 20-12-21 21:24:03 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/odroid-n2-homeassistant with version 2020.12.1 20-12-21 21:24:03 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 9 Home Assistant secrets 20-12-21 21:24:03 INFO (MainThread) [supervisor.hassos] Detect HassOS 5.9 / BootSlot B 20-12-21 21:24:03 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/core repository 20-12-21 21:24:03 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/a0d7b954 repository 20-12-21 21:24:04 INFO (MainThread) [supervisor.store] Loading add-ons from store: 66 all - 66 new - 0 remove 20-12-21 21:24:04 INFO (MainThread) [supervisor.addons] Found 5 installed add-ons 20-12-21 21:24:04 INFO (SyncWorker_2) [supervisor.docker.interface] Attaching to homeassistant/aarch64-addon-samba with version 9.3.0 20-12-21 21:24:04 INFO (SyncWorker_4) [supervisor.docker.interface] Attaching to homeassistant/aarch64-addon-duckdns with version 1.12.1 20-12-21 21:24:04 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/aarch64-addon-ssh with version 8.9.1 20-12-21 21:24:04 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to homeassistant/aarch64-addon-check_config with version 3.6.0 20-12-21 21:24:04 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/aarch64-addon-configurator with version 5.2.0 20-12-21 21:24:04 INFO (MainThread) [supervisor.snapshots] Found 4 snapshot files 20-12-21 21:24:04 INFO (MainThread) [supervisor.discovery] Loaded 0 messages 20-12-21 21:24:04 INFO (MainThread) [supervisor.ingress] Loaded 0 ingress sessions 20-12-21 21:24:04 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.SETUP 20-12-21 21:24:04 INFO (MainThread) [supervisor.resolution.check] System checks complete 20-12-21 21:24:04 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP 20-12-21 21:24:04 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 20-12-21 21:24:04 INFO (MainThread) [main] Running Supervisor 20-12-21 21:24:04 INFO (MainThread) [supervisor.hassos] Rauc: B - marked slot kernel.1 as good 20-12-21 21:24:04 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.INITIALIZE' starting 0 add-ons 20-12-21 21:24:04 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.SYSTEM' starting 0 add-ons 20-12-21 21:24:04 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.SERVICES' starting 3 add-ons 20-12-21 21:24:04 INFO (SyncWorker_2) [supervisor.docker.interface] Cleaning addon_core_samba application 20-12-21 21:24:04 INFO (SyncWorker_2) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-samba with version 9.3.0 20-12-21 21:24:04 INFO (SyncWorker_4) [supervisor.docker.interface] Cleaning addon_core_duckdns application 20-12-21 21:24:05 INFO (SyncWorker_4) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-duckdns with version 1.12.1 20-12-21 21:24:05 INFO (SyncWorker_3) [supervisor.docker.interface] Cleaning addon_core_ssh application 20-12-21 21:24:06 INFO (SyncWorker_3) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-ssh with version 8.9.1 20-12-21 21:24:11 INFO (MainThread) [supervisor.core] Start Home Assistant Core 20-12-21 21:24:11 INFO (SyncWorker_4) [supervisor.docker.interface] Starting homeassistant 20-12-21 21:24:11 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready 20-12-21 21:30:51 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
journalctl
- command not found ;-( -->Kernel logs:
Description of problem: After migration HA from Raspberry PI4 to ODROID-N2 system hangs after few minutes.
installation with emmc image hassos_odroid-n2-5.8.img.xz (system = OK)
Make first snapshot (system = OK)
Upload snapshot file from Raspberry PI 4 with the same versions
Restore snapshot
After restore Core RAM Usage memory usage increases quickly and system hangs after a few minutes
Restore only works when i not restore the folder Home assistant configuration folder (i deselect this folder in restore snapshot properties. The most important folder ;-(
All USB hardware are disconnected at this time.
Problem exists on two devices (a have two seperate HA installations)