Closed elmr91 closed 11 months ago
agners
commented
1 year ago On my machine with KVM hypervisor (which is also used by Proxmox) containerd uses about 0.7% on HAOS 10. But I am running on a rather new AMD CPU.
What type of system are you running on?
Is maybe journalctl -f -u containerd showing something?
elmr91
commented
1 year ago No particular log since boot:
Apr 18 16:22:27 homeassistant containerd[349]: time="2023-04-18T16:22:27.878453230Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/585c29bcc7be293d23741540dbb4cc2af1bd3a22b3685bc4f02f04248e9b9f09 pid=2290 runtime=io.containerd.runc.v2
Apr 18 16:22:33 homeassistant containerd[349]: time="2023-04-18T16:22:33.269643073Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 18 16:22:33 homeassistant containerd[349]: time="2023-04-18T16:22:33.269729478Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 18 16:22:33 homeassistant containerd[349]: time="2023-04-18T16:22:33.269749452Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 18 16:22:33 homeassistant containerd[349]: time="2023-04-18T16:22:33.270792200Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/13b89612a676908649a9c53efe151781e6267309bfde99934415abd8ec969c83 pid=2520 runtime=io.containerd.runc.v2
Apr 18 16:23:09 homeassistant containerd[349]: time="2023-04-18T16:23:09.585064770Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 18 16:23:09 homeassistant containerd[349]: time="2023-04-18T16:23:09.585120961Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 18 16:23:09 homeassistant containerd[349]: time="2023-04-18T16:23:09.585137376Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 18 16:23:09 homeassistant containerd[349]: time="2023-04-18T16:23:09.585665355Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/481308628ffcd074969046225f8a9435aea56255ad7b6d1eb690b2e3f270004e pid=2881 runtime=io.containerd.runc.v2Last log line was 30 minutes ago
agners
commented
1 year ago Also logs in general (using journalctl -f) is there some logging going on?
elmr91
commented
1 year ago Nothing particular:
Apr 18 17:00:40 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:00:51 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:01:01 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:01:12 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:01:22 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:01:33 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:01:43 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:02:23 homeassistant hassio_supervisor[397]: Apr 18 17:03:06 homeassistant qemu-ga[346]: info: guest-ping called Apr 18 17:03:06 homeassistant qemu-ga[346]: info: guest-fsfreeze called
Last line is related to a VM backup I made few minutes ago.
elmr91
commented
1 year ago System is running a low power but quite efficient CPU: 4 x Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz (1 Socket) 24 GB RAM (2 GB allocated for HAOS VM)
elmr91
commented
1 year ago I restored backup taken during last night (HAOS 9.5 / before upgrade) CPU is back to normal (2%)
No containerd process using CPU
wokkeltje13
commented
1 year ago See same behavior but not so big, probably a bit more powerful CPU
The spikes jou see is from the update and corresponding reboot, but after the upgrade you see a slightly higher CPU usage and MEM usage (should be lower after a fresh reboot, but is higher)
dsolva
commented
1 year ago Im using proxmox and also experienced an increase from 6% to 9% on cpu but i dont think I have a more powerful cpu. containerd using 3-4%. Cpu is 2x Intel J4025.
agners
commented
1 year ago Higher RAM usage is somewhat expected as we tuned VM swapiness to use less swap (since we use zswap now, which is stored on the data disk).
As for CPU load: I can't really reproduce this here on my KVM installation, it seems to be about the same on HAOS 9.5 and 10.0: When running top, it appears and disappears from the list, when it appears it is at just 0.7%.
I've also seen that CPU frequency scaling is behaving differently on the new kernel version. Maybe this influences the reported CPU load, not sure. However, since this is virtualization, maybe unlikely :thinking:
evilspoons
commented
1 year ago Similar results here. Host is Proxmox VE 7.4-3, i5-7500. Went from 3.73 GiB to 4.11 GiB RAM (no big deal), CPU went from 2.4-2.5% before upgrade to 3.5% after upgrade.
Virtual machine configuration is 8 GB RAM, 4 processors with "host" as CPU type setting, OVMF (UEFI), i440fx, virtio scsi and LAN, and one USB device passed through.
wokkeltje13
commented
1 year ago Not sure if CPU is more powerful 4 x Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (1 Socket)
Did a rollback
You can see CPU is like before and MEM is a lot lower than before (system had an uptime of 5 days when running 9.5)
So 10.0 after the reboot used 900MB more mem
agners
commented
1 year ago So 10.0 after the reboot used 900MB more mem
As I've said before, this is expected: Home Assistant OS 9.5 used a different swapiness value, meaning it more happily swapped things out (to zram, means part of the memory will get compressed and swapped out). With OS 10.0 it will swap out much later, use more RAM as long as available. Overall, this should not change much, maybe some tasks will now be a bit faster (since the system doesn't need to decompress pages swapped out to zram), or a bit slower (since we work with a slightly smaller page cache), YMMV.
pickonedev
commented
1 year ago CPU and MEMORY 100% usage with the new update...
There is any way to downgrade?
elmr91
commented
1 year ago @evilspoons & @wokkeltje13, we have very similar CPU. I think I know why upgrade from 9.5 to 10 shows a different CPU behavior on my system: I'm using "powersave" CPU governor on host system.
I switched back to "performance" on host :
echo "performance" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
CPU usage on HAOS 10 decreases to 4.5 % (same increase as other have seen):
Using powersave host governor magnifies CPU usage difference between HAOS 9.5 & 10
Upgrade from HAOS 9.5 to 10 increases CPU usage. It has a direct impact on energy usage. My system rose from 6-7 Watt to 8-9 Watt after upgrade
Less than 10W is still a very good value for a hosting system, but HAOS 10 is less energy efficient than 9.5. It seems related to the new docker version.
wokkeltje13
commented
1 year ago mine is also on powersave, did not change during tests with 10.0
pickonedev
commented
1 year ago 
Mine 100% CPU and Memory. Anyone knows how to downgrade the OS version?
dsolva
commented
1 year ago @elmr91 makes sense why your cpu increase seemed greater due to power safe. I have powersafe turned off and the upgrade caused around 0.8watt increase only (i assume because my processor tdp is just 10W). Almost a watt is a lot for my "power efficient" NUC.
elmr91
commented
1 year ago @elmr91 makes sense why your cpu increase seemed greater due to power safe. I have powersafe turned off and the upgrade caused around 0.8watt increase only (i assume because my processor tdp is just 10W). Almost a watt is a lot for my "power efficient" NUC.
mine (i5-7500T) is a 35W TDP processor @wokkeltje13's (i5-6500) and @evilspoons's (i5-7500) are 65W TDP processors They should have also few Watt more on HAOS 10
agners
commented
1 year ago Mine 100% CPU and Memory. Anyone knows how to downgrade the OS version?
Use this command in the command line:
ha os update --version 9.5@agners Thank you!
Any idea why my CPU it is going to full 100 percentage? Memory as well... I use a PC Dell OptiPlex 3090 MFF, Intel® Core™ i7-10700T @4.50 GHz, 32GB DDR4
agners
commented
1 year ago Any idea why my CPU it is going to full 100 percentage? Memory as well... I use a PC Dell OptiPlex 3090 MFF, Intel® Core™ i7-10700T @4.50 GHz, 32GB DDR4
Are you using the motionEye add-on?
Taronaga
commented
1 year ago nuc Intel® Celeron® Processor N3060 after 80% update also problem with mqtt, rolled back to 9.5, loading became 10-12%, MQTT 6.1.3 - normal flight.
p.s. How did the problems start with updating Mosquitto broker to
6.2.0 - I'm already afraid to update, and so I sat without light for half a day
pickonedev
commented
1 year ago Any idea why my CPU it is going to full 100 percentage? Memory as well... I use a PC Dell OptiPlex 3090 MFF, Intel® Core™ i7-10700T @4.50 GHz, 32GB DDR4
Are you using the motionEye add-on?
Yes, I use MotionEye addon as well as other addons.
pickonedev
commented
1 year ago nuc Intel® Celeron® Processor N3060 after 80% update also problem with mqtt, rolled back to 9.5, loading became 10-12%, MQTT 6.1.3 - normal flight. p.s. How did the problems start with updating Mosquitto broker to 6.2.0 - I'm already afraid to update, and so I sat without light for half a day
For me, MQTT 6.20 doesn't have any problem. Only the new OS update made my CPU and MEM going like crazy :-)
elmr91
commented
1 year ago After few hours idling, I can confirm OS10 has a direct impact on host CPU usage (linked to containerd daemon CPU in HAOS VM)
gfn256
commented
1 year ago Running Proxmox on i5-6500T 8gb ram with HASSOS vm on 2 cores 4gb ram Increased VM CPU usage from ver 9.5 to 10.0 more than 3% Proxmox node itself (above machine - all 4 cores) shows increase of about 1% (sometimes even less). Not seeing much changed RAM use on either.
CPU temp on average gone up 2°C (to 39°C) - This could probably be used to show increased energy cost!
Also of interest, the upgrade was completely seamless, with NO errors what so ever, the whole thing took maybe 1-2 minutes.
Edit: After running system for about 8 hours, I'm actually observing an increased temp average of about 5°C (to 42°C), this in my book is WASTED energy, and SHOULD be addressed. Other stats seem as above in OP. I'll also note, that as a whole I find the system snappier and generally faster, such as boot-up time, web browser response etc.. This is welcome! Thanks team.
pickonedev
commented
1 year ago Maybe the next update, will have some better optimisations...
danielholm
commented
1 year ago My unit went from 20% to 100%. I disabled some docker containers and now it's a bit lower than before, but still much higher spikes than using 9.5. Very smooth upgrade, though.
Galapalos
commented
1 year ago Hi,
After upgrade, my HA prosessor use went from 30% --> 100%, some thin is not right
here info about prosessor use and memory before and after upgrade
Here is system information if it helps
| version | core-2023.4.5 |
|---|---|
| installation_type | Home Assistant OS |
| dev | false |
| hassio | true |
| docker | true |
| user | root |
| virtualenv | false |
| python_version | 3.10.10 |
| os_name | Linux |
| os_version | 6.1.21-v8 |
| arch | aarch64 |
| timezone | Europe/Helsinki |
| config_dir | /config |
elmr91
commented
1 year ago I tried to simply restart containerd service:
systemctl restart containerd
CPU used by containerd was divided by 3 after service restart (from 7-8% to less than 2%). Host CPU dropped from 6-7 to 4-5 %
Edit: CPU used by containerd returned slowly to its initial value 6-7% after one hour.
Some error message related to RO filesystem show during initial service start at boot are not shown anymore while restarting containerd service:
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622559719Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt: read-only file system"
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622640728Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622716682Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622778815Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622841541Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622902489Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.622960860Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.623040770Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Apr 19 11:28:52 homeassistant containerd[353]: time="2023-04-19T11:28:52.623220915Z" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1@Galapalos @danielholm @pickonedev the very high CPU load is most likely related to motionEye. Frenck is working on an upgrade, it should be available soon. However, this is kinda off-topic here, as this issue is about a a slightly higher CPU usage due to containerd. If the issue persists after the motionEye update, please open a new issue so we can track it separately.
geobogb
commented
1 year ago I'm not running motionEye but have the same problem with high CPU usage on 10.0. I'm running Home Assistant in a headless Virtual Box machine. On 9.5 it uses around 5% CPU on the host machine (Debian/bullseye). After upgrading to 10.0. the CPU usage rose to over 25%. After downgrading to 9.5, everything is back to normal.
Please let me know if I can provide any help in resolving this issue!
agners
commented
1 year ago @Galapalos @danielholm @pickonedev @geobogb please make sure to run motionEye 0.19.0, it should resolve the high CPU usage problem by that add-on. If not, please disable the add-on and see if something else is using more CPU. If it is down to motionEye, we should continue tracking the problem in motionEye's issue tracker: https://github.com/hassio-addons/addon-motioneye/issues.
gfn256
commented
1 year ago Thought I'd share my CPU temps graph since updating to 10.0 - and I don't like the look of it, see my post above - I'm going to get my Tasmota energy measuring socket out to check the wattage usage now.
Info is sent from Proxmox VE by mqtt every 5 mins (cronjob) to HA I marked for reference, the 2 spikes - which are the nightly PVE VM HAOS backups. Checked the ambient temp and its pretty stable for period.
geobogb
commented
1 year ago @Galapalos @danielholm @pickonedev @geobogb please make sure to run motionEye 0.19.0,
But I'm not using/running motionEye at all!
andrazek
commented
1 year ago I experience the same problem. My temps went up by about 5°C with this upgrade. For passive mini PCs that's definitely too much. It will wear out sooner. How can we find the cause of this problem? I'm not using motionEye.
pickonedev
commented
1 year ago @agners I made the new update for motionEye and OS and everything is ok now. The CPU and MEM have the same values as before the issue :-) Thank you!
lyricnz
commented
1 year ago I have the same problem as reported multiple times above. High CPU usage since 10.0, VM hosted on proxmox, with plenty of CPU + RAM available. Screenshots of HA and proxmox below
CPU usage appears to be in homeassistant container (according to "top" and "docker stats"). I'm not running motioneye.
lyricnz
commented
1 year ago Reverting to 9.5 caused an immediate lowering of CPU usage
FWIW I don't appear to be using swap at all
➜ ~ free
total used free shared buff/cache available
Mem: 2472664 755520 841720 5132 875424 1683284
Swap: 618164 0 618164
SeByDocKy
commented
1 year ago For me I jumped from 6% to 30% (Hassio Odroid C4)
SeByDocKy
commented
1 year ago @Galapalos @danielholm @pickonedev the very high CPU load is most likely related to motionEye. Frenck is working on an upgrade, it should be available soon. However, this is kinda off-topic here, as this issue is about a a slightly higher CPU usage due to containerd. If the issue persists after the motionEye update, please open a new issue so we can track it separately.
I even uninstall motioneye and still x 5 more usage.... for me it's not related....
Edit : I returned to 9.5 everything returned to normal ... I will wait for the 10.x :) fix
pickonedev
commented
1 year ago For me it is even better than before... In the past, CPU was 4%, MEM 8% and TEMP 38 C ... After the OS update, I had problems with CPU & MEM at 100%, but after I have updated MotionEye and again OS, the CPU is at 3%, mem at 7,2% and temp at 32 C
....
jlunz
commented
1 year ago I am seeing a similar behavior on a Raspberry Pi Zero 2 W (I know, not recommended ;-)) Load (15m) jumped from almost zero to ~0.5
geobogb
commented
1 year ago Still same problem running Home Assistant in Virtual Box. Upgraded to 10.0 around 15:30 and reverted back to 9.5 around 18:30. From the inside, the CPU increase might not be so dramatic:
But seen from the host it is!
And, with the risk of repeating myself, I'm not using motionEye.
wokkeltje13
commented
1 year ago I think there are enough people who showed evidence that version 10 has an impact on resource usage. Increased mem usage is by design, but I do not see any reason why CPU and related to that heat and power consumption should be higher (The MotionEye thing is not relevant to this issue since most people even don't have it installed).
I also see no steps in troubleshooting or remediation appear. Is there something we can do to help since most people just like I did, returned to 9.5. But this is also no long-term solution.
lyricnz
commented
1 year ago Nothing in current 10.1 beta appears to relate to this issue https://github.com/home-assistant/operating-system/releases/tag/10.1
lyricnz
commented
1 year ago I tested upgrade, followed by some usage, followed by HAOS revert. CPU vs versions.
HA core version, and all integrations, remain unchanged.
jandado
commented
1 year ago I can confirm on my somewhat new AMD Ryzen 7 5800H, the VM has 2 cores allocated to it. CPU usage doubled, containerd appears to be behind it.
dsolva
commented
1 year ago OS 10.1 oficial release is out. I'm curious if the problem persists.
jandado
commented
1 year ago Yes it does persist @dsolva. I have upgraded over an hour ago, I see no change compared to 10.0
Describe the issue you are experiencing
I have juste upgraded my proxmox HAOS VM to OS 10 I immediately noticed CPU usage raising from around 2% to 10% after upgrade.
"docker stats" shows a normal container usage / nearly no load.
"top" shows containerd is using a consistent 6-8% CPU (this is the only process using significant CPU load)
I rebooted the VM, but CPU load stays the same:
What operating system image do you use?
ova (for Virtual Machines)
What version of Home Assistant Operating System is installed?
10
Did you upgrade the Operating System.
Yes
Steps to reproduce the issue
1.Install 9.5 ova image in proxmox 2.Upgrade to Operating System 10 3. ...
Anything in the Supervisor logs that might be useful for us?
Anything in the Host logs that might be useful for us?
System information
`## System Information
Home Assistant Community Store
GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4975 Installed Version | 1.32.1 Stage | running Available Repositories | 1274 Downloaded Repositories | 3Home Assistant Cloud
logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | okHome Assistant Supervisor
host_os | Home Assistant OS 10.0 -- | -- update_channel | stable supervisor_version | supervisor-2023.04.0 agent_version | 1.5.1 docker_version | 23.0.3 disk_total | 30.8 GB disk_used | 3.9 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons | Terminal & SSH (9.6.1), File editor (5.5.0)Dashboards
dashboards | 2 -- | -- resources | 1 views | 5 mode | storageRecorder
oldest_recorder_run | 12 avril 2023 à 19:39 -- | -- current_recorder_run | 18 avril 2023 à 18:22 estimated_db_size | 179.41 MiB database_engine | sqlite database_version | 3.38.5 [