search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
69.87k stars 28.97k forks source link

Home Assistant Slow with increased CPU usage #121327

Open Axsgranted opened 5 days ago

Axsgranted commented 5 days ago

The problem

Problem: Recently updated to the latest Core of 2024.7.x and noticed Home Assistant frontend was slow to load pages, where previously everything was snappy. I've run into this issue with home assistant once before on the same hardware. An update ultimately fixed it.

Observations:

  1. Found CPU usage to be in the 30-40+% range where previously it was running 7% to 15%. Checked to see if the issue was related to a 3rd party or 1st party integration. Restarted in "safe mode" which reduced usage by only 10% and responsiveness was still slow and laggy.

Fig 1. (Everything enabled) image

Fig 2. (Safe Mode) image

  1. I then investigated to see if it was a hardware issue. I used Glance and CPU Speed to check was running and potentially hogging resources. CPU showed running at normal frequency and Glances showed the home assistant container using all the resources. PLEASE NOTE: In the glances image I noticed VERY high cpu context switching (ctx_sw).

Fig. 3 (Glances Snapshot) image

Fig 4. (CPU Speed) image

What version of Home Assistant Core has the issue?

2024.7.1

What was the last working version of Home Assistant Core?

Previous to 2024.7.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

None

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

bdraco commented 5 days ago

Please follow this guide

https://community.home-assistant.io/t/2024-5-tracking-down-instability-issues-caused-by-integrations/724441

Sluggish performance or increased CPU usage

Axsgranted commented 5 days ago

Image below. What view should I provide?

image

bdraco commented 5 days ago

Please post the callgrind file

Axsgranted commented 5 days ago

callgrind.out.zip

See attached!

bdraco commented 5 days ago

It looks like something is doing a massive amount of listdir calls in the event loop

bdraco commented 5 days ago

Do you have warnings about the event loop being blocked in the log?

Axsgranted commented 5 days ago

Do you have warnings about the event loop being blocked in the log?

I don't block anything in the logs.

bdraco commented 5 days ago

Can you try turning on debug logging and see if the log is flooded with errors about the event loop being blocked?

Axsgranted commented 5 days ago

I know how to turn on debugging for specific integrations, but I assume you mean all logs. Is this the appropriate guidance for that?

https://www.home-assistant.io/integrations/logger/

bdraco commented 5 days ago

Try these first:

# Example configuration.yaml entry
logger:
  default: info
  logs:
    homeassistant.util.loop: debug
    homeassistant.helpers.frame: debug

If that doesn't show anything, you can try turning on debug for everytime

# Example configuration.yaml entry
logger:
  default: debug
Axsgranted commented 5 days ago

Only log with event loop is pasted below but only showing 10 times. I'm enabling debug for default for all logs now and will check again.

Logger: homeassistant.helpers.frame Source: helpers/frame.py:184 First occurred: 1:45:18 PM (10 occurrences) Last logged: 1:46:20 PM

Detected that custom integration 'hacs' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call await hass.http.async_register_static_paths([StaticPathConfig("/hacsfiles", "/config/www/community", True)]); This function will be removed in 2025.7 at custom_components/hacs/base.py, line 1162: self.hass.http.register_static_path(, please create a bug report at https://github.com/hacs/integration/issues Detected that custom integration 'meross_cloud' calls async_forward_entry_setup for integration, meross_cloud with title: https://iot.meross.com and entry_id: 328616ab32bc843fab893e6674abb551, which is deprecated and will stop working in Home Assistant 2025.6, await async_forward_entry_setups instead at custom_components/meross_cloud/init.py, line 427: hass.async_create_task(, please create a bug report at https://github.com/albertogeniola/meross-homeassistant Detected that custom integration 'dahua' calls async_forward_entry_setup for integration, dahua with title: Front Doorbell - Dahua and entry_id: 876509d754abb7e121af8325ea41f607, which is deprecated and will stop working in Home Assistant 2025.6, await async_forward_entry_setups instead at custom_components/dahua/init.py, line 85: hass.async_create_task(, please create a bug report at https://github.com/rroller/dahua/issues Detected that custom integration 'bambu_lab' calls async_forward_entry_setup for integration, bambu_lab with title: 00M09A391300052 and entry_id: beb32588b9443b78b67f0f4cfe13bd8c, which is deprecated and will stop working in Home Assistant 2025.6, await async_forward_entry_setups instead at custom_components/bambu_lab/coordinator.py, line 196: await self.hass.config_entries.async_forward_entry_setup(self.config_entry, Platform.SENSOR), please create a bug report at https://github.com/greghesp/ha-bambulab/issues Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues

bdraco commented 5 days ago

Please download the raw log and check it instead of looking at the UI

Axsgranted commented 5 days ago

Found this event loop listed a massive amount of times. going to disable the integration to see if anything changes.

2024-07-05 13:51:00.490 DEBUG (MainThread) [homeassistant.util.loop] Detected blocking call to listdir with args ('/usr/local/lib/python3.12/site-packages/botocore/data/appconfigdata',) inside the event loop by custom integration 'petsafe' at custom_components/petsafe/init.py, line 244: self._feeders = await self.api.get_feeders() (offender: /usr/local/lib/python3.12/site-packages/botocore/loaders.py, line 311: api_versions = os.listdir(full_dirname)), please create a bug report at https://github.com/dcmeglio/homeassistant-petsafe/issues For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#listdir

Axsgranted commented 5 days ago

Disabling the petsafe integration appears to have had an impact. UI is very responsive, CPU usage has dropped to 30 ish percent. Will keep an eye on it and report back if anything changes!

What is odd is that I ran Home assistant in safe mode which should have disabled that no? And I was still seeing the behavior.

brwyatt commented 4 days ago

Had the same issue here after upgrading to 2024.7.0 (and 2024.7.1) from 2024.6.x. Also had the PetSafe integration installed. Disabling the integration similarly took the CPU utilization from the elevated 26-33% back down to the pre-upgrade level of 4-12% (on Home Assistant Yellow).

So definitely appears to be some bad interaction between that integration and something that changed in this release.

brwyatt commented 4 days ago

Cut an issue over on the PetSafe integration repo about this.