search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.13k stars 29.8k forks source link

Higher CPU continues in 0.118 #40292

Closed gieljnssns closed 3 years ago

gieljnssns commented 3 years ago

The problem

I don't know if it is a core or frontend problem or something else. But I'm seeing a higher CPU use in generally, but especially when clicking on a card in the Lovelace view. My normal CPU use was 2 a 3 % now it is at 6 to 8 % When I want to show the history the fans of my NUC immediately start to blow and I see always a turning circle.

Schermafbeelding 2020-09-19 om 10 24 13

And the CPU use peaks at about 18%

This are some screenshots from Glances when clicking on history in Lovelace Schermafbeelding 2020-09-19 om 10 30 15 Schermafbeelding 2020-09-19 om 10 30 20 Schermafbeelding 2020-09-19 om 10 30 29

I had debug on for

logs:
  homeassistant.event: debug
  homeassistant.components.logger: debug
  homeassistant.components.history: debug
  homeassistant.components.recorder: debug

But not seeing anything special

I've tried the Py-Spy way, but I can not get it installed

   Compiling addr2line v0.11.0
     Running `rustc --crate-name addr2line /root/.cargo/registry/src/github.com-1ecc6299db9ec823/addr2line-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="cpp_demangle"' --cfg 'feature="default"' --cfg 'feature="object"' --cfg 'feature="rustc-demangle"' --cfg 'feature="std"' --cfg 'feature="std-object"' -C metadata=361484a7c1298d46 -C extra-filename=-361484a7c1298d46 --out-dir /tmp/cargo-installWui9CK/release/deps -L dependency=/tmp/cargo-installWui9CK/release/deps --extern cpp_demangle=/tmp/cargo-installWui9CK/release/deps/libcpp_demangle-eb3cfddebaa342f3.rmeta --extern fallible_iterator=/tmp/cargo-installWui9CK/release/deps/libfallible_iterator-a4123992588425b5.rmeta --extern gimli=/tmp/cargo-installWui9CK/release/deps/libgimli-7d5edb096fedfa90.rmeta --extern lazycell=/tmp/cargo-installWui9CK/release/deps/liblazycell-ea0ac196d0b3629c.rmeta --extern object=/tmp/cargo-installWui9CK/release/deps/libobject-477a467cbf075a12.rmeta --extern rustc_demangle=/tmp/cargo-installWui9CK/release/deps/librustc_demangle-1fe779301a7b7227.rmeta --extern smallvec=/tmp/cargo-installWui9CK/release/deps/libsmallvec-9bd83c2162524722.rmeta --cap-lints allow`
     Running `rustc --crate-name remoteprocess /root/.cargo/registry/src/github.com-1ecc6299db9ec823/remoteprocess-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unwind"' -C metadata=2427f6658e4e467c -C extra-filename=-2427f6658e4e467c --out-dir /tmp/cargo-installWui9CK/release/deps -L dependency=/tmp/cargo-installWui9CK/release/deps --extern addr2line=/tmp/cargo-installWui9CK/release/deps/libaddr2line-361484a7c1298d46.rmeta --extern benfred_read_process_memory=/tmp/cargo-installWui9CK/release/deps/libbenfred_read_process_memory-e58b01e0194ad895.rmeta --extern goblin=/tmp/cargo-installWui9CK/release/deps/libgoblin-f7d9a003d2b2b860.rmeta --extern lazy_static=/tmp/cargo-installWui9CK/release/deps/liblazy_static-dda545747076ca2e.rmeta --extern libc=/tmp/cargo-installWui9CK/release/deps/liblibc-d6a517a54d66530e.rmeta --extern log=/tmp/cargo-installWui9CK/release/deps/liblog-33247e2f897fd775.rmeta --extern memmap=/tmp/cargo-installWui9CK/release/deps/libmemmap-d558b4e09fe43210.rmeta --extern nix=/tmp/cargo-installWui9CK/release/deps/libnix-003b039a35e9d039.rmeta --extern object=/tmp/cargo-installWui9CK/release/deps/libobject-477a467cbf075a12.rmeta --extern proc_maps=/tmp/cargo-installWui9CK/release/deps/libproc_maps-af79a59c980f3ade.rmeta --extern regex=/tmp/cargo-installWui9CK/release/deps/libregex-bc89e9f6ea47428c.rmeta --cap-lints allow -L native=/usr/local/lib -l static=unwind -l static=unwind-ptrace -l static=unwind-x86_64`
     Running `rustc --crate-name py_spy /root/.cargo/registry/src/github.com-1ecc6299db9ec823/py-spy-0.3.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ff12da3be12bb5c3 -C extra-filename=-ff12da3be12bb5c3 --out-dir /tmp/cargo-installWui9CK/release/deps -L dependency=/tmp/cargo-installWui9CK/release/deps --extern clap=/tmp/cargo-installWui9CK/release/deps/libclap-f6bf72e393cc38bc.rmeta --extern console=/tmp/cargo-installWui9CK/release/deps/libconsole-f112d37c0f47569f.rmeta --extern cpp_demangle=/tmp/cargo-installWui9CK/release/deps/libcpp_demangle-eb3cfddebaa342f3.rmeta --extern ctrlc=/tmp/cargo-installWui9CK/release/deps/libctrlc-aad786bac40b3a9d.rmeta --extern env_logger=/tmp/cargo-installWui9CK/release/deps/libenv_logger-e27ea58ddced9d00.rmeta --extern failure=/tmp/cargo-installWui9CK/release/deps/libfailure-3335c4e883f54b39.rmeta --extern goblin=/tmp/cargo-installWui9CK/release/deps/libgoblin-f7d9a003d2b2b860.rmeta --extern indicatif=/tmp/cargo-installWui9CK/release/deps/libindicatif-225faf62a9b3f910.rmeta --extern inferno=/tmp/cargo-installWui9CK/release/deps/libinferno-d5bd4adb491d743b.rmeta --extern lazy_static=/tmp/cargo-installWui9CK/release/deps/liblazy_static-dda545747076ca2e.rmeta --extern libc=/tmp/cargo-installWui9CK/release/deps/liblibc-d6a517a54d66530e.rmeta --extern log=/tmp/cargo-installWui9CK/release/deps/liblog-33247e2f897fd775.rmeta --extern lru=/tmp/cargo-installWui9CK/release/deps/liblru-7c22ae3502192a2e.rmeta --extern memmap=/tmp/cargo-installWui9CK/release/deps/libmemmap-d558b4e09fe43210.rmeta --extern proc_maps=/tmp/cargo-installWui9CK/release/deps/libproc_maps-af79a59c980f3ade.rmeta --extern rand=/tmp/cargo-installWui9CK/release/deps/librand-3b4eb2183b20dcb0.rmeta --extern rand_distr=/tmp/cargo-installWui9CK/release/deps/librand_distr-9984e8ecbf04f38f.rmeta --extern regex=/tmp/cargo-installWui9CK/release/deps/libregex-bc89e9f6ea47428c.rmeta --extern remoteprocess=/tmp/cargo-installWui9CK/release/deps/libremoteprocess-2427f6658e4e467c.rmeta --extern serde=/tmp/cargo-installWui9CK/release/deps/libserde-64b415a5891575a4.rmeta --extern serde_derive=/tmp/cargo-installWui9CK/release/deps/libserde_derive-87637e81ef141bef.so --extern serde_json=/tmp/cargo-installWui9CK/release/deps/libserde_json-8b40eb8bd5b4f85c.rmeta --extern tempfile=/tmp/cargo-installWui9CK/release/deps/libtempfile-2085f7c84ee47d2e.rmeta --extern termios=/tmp/cargo-installWui9CK/release/deps/libtermios-776eb0ed1fc2710a.rmeta --cap-lints allow --cfg unwind -L native=/usr/local/lib`
error: could not find native static library `unwind`, perhaps an -L flag is missing?

error: aborting due to previous error

error: could not compile `remoteprocess`.

Caused by:
  process didn't exit successfully: `rustc --crate-name remoteprocess /root/.cargo/registry/src/github.com-1ecc6299db9ec823/remoteprocess-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unwind"' -C metadata=2427f6658e4e467c -C extra-filename=-2427f6658e4e467c --out-dir /tmp/cargo-installWui9CK/release/deps -L dependency=/tmp/cargo-installWui9CK/release/deps --extern addr2line=/tmp/cargo-installWui9CK/release/deps/libaddr2line-361484a7c1298d46.rmeta --extern benfred_read_process_memory=/tmp/cargo-installWui9CK/release/deps/libbenfred_read_process_memory-e58b01e0194ad895.rmeta --extern goblin=/tmp/cargo-installWui9CK/release/deps/libgoblin-f7d9a003d2b2b860.rmeta --extern lazy_static=/tmp/cargo-installWui9CK/release/deps/liblazy_static-dda545747076ca2e.rmeta --extern libc=/tmp/cargo-installWui9CK/release/deps/liblibc-d6a517a54d66530e.rmeta --extern log=/tmp/cargo-installWui9CK/release/deps/liblog-33247e2f897fd775.rmeta --extern memmap=/tmp/cargo-installWui9CK/release/deps/libmemmap-d558b4e09fe43210.rmeta --extern nix=/tmp/cargo-installWui9CK/release/deps/libnix-003b039a35e9d039.rmeta --extern object=/tmp/cargo-installWui9CK/release/deps/libobject-477a467cbf075a12.rmeta --extern proc_maps=/tmp/cargo-installWui9CK/release/deps/libproc_maps-af79a59c980f3ade.rmeta --extern regex=/tmp/cargo-installWui9CK/release/deps/libregex-bc89e9f6ea47428c.rmeta --cap-lints allow -L native=/usr/local/lib -l static=unwind -l static=unwind-ptrace -l static=unwind-x86_64` (exit code: 1)
warning: build failed, waiting for other jobs to finish...
error: failed to compile `py-spy v0.3.3`, intermediate artifacts can be found at `/tmp/cargo-installWui9CK`

Caused by:
  build failed
bash-5.0# # pip install py-spy
/bin/ash: pip: not found

Environment

arch x86_64
chassis embedded
dev false
docker true
docker_version 19.03.11
hassio true
host_os HassOS 4.13
installation_type Home Assistant OS
os_name Linux
os_version 5.4.63
python_version 3.8.5
supervisor 245
timezone Europe/Brussels
version 0.115.1
virtualenv false 

## Problem-relevant `configuration.yaml`
<!--
  An example configuration that caused the problem for you. Fill this out even
  if it seems unimportant to you. Please be sure to remove personal information
  like passwords, private URLs and other credentials.
-->

```yaml
I don't know

Traceback/Error logs

None

Additional information

If you need some more info, ask...

gieljnssns commented 3 years ago

When not using the frontend, once a 50-60 second the fans of my NUC are blowing fast. This is a screenshot of glances the moment this happens. Schermafbeelding 2020-09-19 om 11 25 48

springstan commented 3 years ago

Could be related to this issue #39890

bdraco commented 3 years ago

There isn't much we can do to help without a py-spy or interesting logs.

gieljnssns commented 3 years ago

Maybe you can tell me how to install....

gieljnssns commented 3 years ago

I've tried https://developers.home-assistant.io/docs/operating-system/debugging/#ssh-access-to-the-host and then

apk add cargo
cargo install py-spy
gieljnssns commented 3 years ago

There isn't much we can do to help without a py-spy or interesting logs.

Which Debug you need? I've tried

logs:
  homeassistant.event: debug
  homeassistant.components.logger: debug
  homeassistant.components.history: debug
  homeassistant.components.recorder: debug
bdraco commented 3 years ago

Try turning on full debug mode. (default: debug) and seeing if there is anything interesting. Without a py-spy to narrow the issue down, its not possible to give any more guidance for which area to look.

gieljnssns commented 3 years ago

Is there a way to send you this debug log? Btw I've tried another 2 times to install Py-Spy, always

error: could not find native static library `unwind`, perhaps an -L flag is missing?

error: aborting due to previous error

error: could not compile `remoteprocess`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `py-spy v0.3.3`, intermediate artifacts can be found at `/tmp/cargo-installNtWBUG`

Caused by:
  build failed
bdraco commented 3 years ago

If you want to send it privately, nick@koston.org would be fine.

Sent from my Mobile

On Sep 19, 2020, at 9:11 AM, Giel Janssens notifications@github.com wrote:

 Is there a way to send you this debug log?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

gieljnssns commented 3 years ago

I've send you a wetransfer link because .log is 100MB

bdraco commented 3 years ago 
2020-09-19 16:16:05 DEBUG (MainThread) [homeassistant.helpers.event] Template update {% if states('sensor.uptime_in_uren')|float > 0.05|float %}
  {{states|selectattr('state', 'in', ['unavailable','unknown','none'])
    |reject('in', expand('group.entity_blacklist'))
    |reject('eq', states.group.entity_blacklist)
    |list|length}}
{% else %}
  0
{% endif %}

Try disabling this template as its watching all states and you have a massive amount of state changed events happening.

You could replace it with an automation to only happen every few seconds and it should help

bdraco commented 3 years ago

Also change

{{ states.sensor | list | length }} to {{ states.sensor | count }}

This will avoid creating a list every time a sensor updates

gieljnssns commented 3 years ago 
2020-09-19 16:16:05 DEBUG (MainThread) [homeassistant.helpers.event] Template update {% if states('sensor.uptime_in_uren')|float > 0.05|float %}
  {{states|selectattr('state', 'in', ['unavailable','unknown','none'])
    |reject('in', expand('group.entity_blacklist'))
    |reject('eq', states.group.entity_blacklist)
    |list|length}}
{% else %}
  0
{% endif %}

I've had tried that yesterday, there was no difference. So I turned it back on...

But I will make an automation of that...

bdraco commented 3 years ago

{{ states.device_tracker | list | length }} as well

bdraco commented 3 years ago

I can tell you what I see is inefficient, but I'm pretty much shooting in the dark without a py-spy.

gieljnssns commented 3 years ago

I'll try to change that all. Py-Spy, I did try to install another 2 times....

bdraco commented 3 years ago 
bdraco@Js-MacBook-Pro-3 ~ % grep 'state_changed' home-assistant\ \(1\).log | grep '16:14:40'|wc
     315   10547  187923
bdraco@Js-MacBook-Pro-3 ~ % grep 'state_changed' home-assistant\ \(1\).log | grep '16:14:41'|wc
     241    7817  135811
bdraco@Js-MacBook-Pro-3 ~ % grep 'state_changed' home-assistant\ \(1\).log | grep '16:14:42'|wc
     355   16331  251219
bdraco@Js-MacBook-Pro-3 ~ % grep 'state_changed' home-assistant\ \(1\).log | grep '16:14:43'|wc
     259    8447  146329
bdraco@Js-MacBook-Pro-3 ~ % grep 'state_changed' home-assistant\ \(1\).log | grep '16:14:44'|wc
     291    9860  176954

It looks like you are getting about 300 state changed events per second 😮

bdraco commented 3 years ago

I'd focus on reducing the state change events to only what you need.

gieljnssns commented 3 years ago

I'd focus on reducing the state change events to only what you need.

How do you mean? Delete some sensors?

In 0.114.x I haven't seen these problems...

gieljnssns commented 3 years ago

I have disabled my unavailable sensor and replaced all | list | length }} by | count }} and

When not using the frontend, once a 50-60 second the fans of my NUC are blowing fast.

this appears to be less frequent

gieljnssns commented 3 years ago

I have found this

2020-09-19 16:42:46 23 [Warning] Aborted connection 23 to db: 'homeassistant' user: 'hass' host: '172.30.32.1' (Got an error reading communication packets)
2020-09-19 17:58:12 35 [Warning] Aborted connection 35 to db: 'homeassistant' user: 'hass' host: '172.30.32.1' (Got an error reading communication packets)

in the logs from the MariaDB addon

bdraco commented 3 years ago

I'd focus on reducing the state change events to only what you need.

How do you mean? Delete some sensors?

In 0.114.x I haven't seen these problems...

If you disable all your template entities, does the issue go away?

gieljnssns commented 3 years ago

I think I removed all template sensors and template binary sensors The restart off HA takes still a lot of time Schermafbeelding 2020-09-19 om 18 34 52 CPU is still higher then 0.114.x And when I click something in the frontend Schermafbeelding 2020-09-19 om 18 36 13

So I restore all my templates...

bdraco commented 3 years ago

It is likely the mysql load is caused by the processing of all those state changed events.

I'm not sure what has changed between 0.114 and 0.115 that would cause that increase. It may be caused by a change in the integration that is generating these events.

amelchio commented 3 years ago

error: could not find native static library unwind, perhaps an -L flag is missing?

I get the same error with cargo. However, this worked for me inside a 0.115.1 Container install:

echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py
pip install py-spy
py-spy record --pid $(pidof python3) --duration 120 --output /config/www/spy.svg

(I did have to add the SYS_PTRACE capability to the container but I assume this is handled already in the OS setup.)

gieljnssns commented 3 years ago

https://www.dropbox.com/s/elvi42f589055gs/spy.svg?dl=0

gieljnssns commented 3 years ago
Schermafbeelding 2020-09-20 om 12 13 41
gieljnssns commented 3 years ago 
bash-5.0# py-spy dump --pid 232
Process 232: python3 -m homeassistant --config /config
Python v3.8.5 (/usr/local/bin/python3.8)

Thread 232 (idle): "MainThread"
    select (selectors.py:468)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    run_until_complete (asyncio/base_events.py:603)
    run (asyncio/runners.py:43)
    run (homeassistant/runner.py:133)
    main (homeassistant/__main__.py:312)
    <module> (homeassistant/__main__.py:320)
    _run_code (runpy.py:87)
    _run_module_as_main (runpy.py:194)
Thread 254 (idle): "Thread-1"
    dequeue (logging/handlers.py:1427)
    _monitor (logging/handlers.py:1478)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 255 (idle): "SyncWorker_0"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 256 (idle): "SyncWorker_1"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 257 (idle): "SyncWorker_2"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 258 (idle): "SyncWorker_3"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 259 (idle): "SyncWorker_4"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 260 (idle): "SyncWorker_5"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 263 (idle): "SyncWorker_6"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 264 (idle): "SyncWorker_7"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 265 (idle): "SyncWorker_8"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 266 (idle): "SyncWorker_9"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 267 (idle): "SyncWorker_10"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 268 (idle): "SyncWorker_11"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 269 (idle): "SyncWorker_12"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 270 (idle): "SyncWorker_13"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 271 (idle): "SyncWorker_14"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 272 (idle): "SyncWorker_15"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 273 (idle): "SyncWorker_16"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 274 (idle): "SyncWorker_17"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 275 (idle): "SyncWorker_18"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 276 (idle): "SyncWorker_19"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 277 (idle): "SyncWorker_20"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 278 (idle): "SyncWorker_21"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 279 (idle): "SyncWorker_22"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 280 (idle): "Recorder"
    run (homeassistant/components/recorder/__init__.py:346)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 281 (idle): "Thread-2"
    dequeue (logging/handlers.py:1427)
    _monitor (logging/handlers.py:1478)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 284 (idle): "zeroconf-Engine-284"
    run (zeroconf/__init__.py:1325)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 285 (idle): "zeroconf-Reaper_285"
    wait (threading.py:306)
    run (zeroconf/__init__.py:1451)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 286 (idle): "SyncWorker_23"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 287 (idle): "SyncWorker_24"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 288 (idle): "SyncWorker_25"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 289 (idle): "SyncWorker_26"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 290 (idle): "SyncWorker_27"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 291 (idle): "SyncWorker_28"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 292 (idle): "SyncWorker_29"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 294 (idle): "SyncWorker_30"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 295 (idle): "SyncWorker_31"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 296 (idle): "SyncWorker_32"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 297 (idle): "SyncWorker_33"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 298 (idle): "SyncWorker_34"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 299 (idle): "SyncWorker_35"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 300 (idle): "SyncWorker_36"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 301 (idle): "SyncWorker_37"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 302 (idle): "SyncWorker_38"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 303 (idle): "SyncWorker_39"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 304 (idle): "SyncWorker_40"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 305 (idle): "SyncWorker_41"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 306 (idle): "SyncWorker_42"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 307 (idle): "SyncWorker_43"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 308 (idle): "SyncWorker_44"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 309 (idle): "SyncWorker_45"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 310 (idle): "SyncWorker_46"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 311 (idle): "SyncWorker_47"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 312 (idle): "SyncWorker_48"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 313 (idle): "SyncWorker_49"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 314 (idle): "SyncWorker_50"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 315 (idle): "SyncWorker_51"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 320 (idle): "Thread-5"
    loop (paho/mqtt/client.py:1163)
    loop_forever (paho/mqtt/client.py:1782)
    _thread_main (paho/mqtt/client.py:3428)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 326 (idle): "Thread-6"
    select (selectors.py:415)
    handle_request (socketserver.py:294)
    run (pysonos/events.py:149)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 327 (idle): "Thread-7"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 328 (idle): "Thread-8"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 329 (idle): "Thread-9"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 331 (idle): "Thread-11"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 334 (idle): "Thread-14"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 336 (idle): "Thread-16"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 337 (idle): "Thread-17"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 340 (idle): "Thread-20"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 342 (idle): "Thread-22"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 343 (idle): "Thread-23"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 344 (idle): "Thread-24"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 346 (idle): "Thread-26"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 347 (idle): "Thread-27"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 348 (idle): "Thread-28"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 349 (idle): "Thread-29"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 353 (idle): "Thread-33"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 354 (idle): "Thread-34"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 356 (idle): "Thread-36"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 358 (idle): "Thread-38"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 360 (idle): "Thread-40"
    wait (threading.py:306)
    wait (threading.py:558)
    run (pysonos/events.py:362)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 377 (idle): "SyncWorker_52"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 378 (idle): "SyncWorker_53"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 379 (idle): "SyncWorker_54"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 380 (idle): "SyncWorker_55"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 381 (idle): "SyncWorker_56"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 382 (idle): "SyncWorker_57"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 383 (idle): "SyncWorker_58"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 384 (idle): "SyncWorker_59"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 385 (idle): "SyncWorker_60"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 386 (idle): "SyncWorker_61"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 387 (idle): "SyncWorker_62"
    query (MySQLdb/connections.py:259)
    _query (MySQLdb/cursors.py:319)
    execute (MySQLdb/cursors.py:206)
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1276)
    _execute_clauseelement (sqlalchemy/engine/base.py:1124)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3533)
    __iter__ (sqlalchemy/orm/query.py:3508)
    yield_events (homeassistant/components/logbook/__init__.py:418)
    humanify (homeassistant/components/logbook/__init__.py:251)
    _get_events (homeassistant/components/logbook/__init__.py:516)
    json_events (homeassistant/components/logbook/__init__.py:228)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 388 (idle): "SyncWorker_63"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 827 (idle): "zeroconf-ServiceBrowser__printer._tcp.local.-_nut._tcp.local.-_miio._udp.local.-_esphomelib._tcp.local.-_ipps._tcp.local.-_plugwise._tcp.local.-_ssh._tcp.local.-_Volumio._tcp.local.-_elg._tcp.local.-_ipp._tcp.local.-_bond._tcp.local.-_googlecast._tcp.local.-_http._tcp.local.-_hap._tcp.local.-_axis-video._tcp.local.-_dkapi._tcp.local.-_api._udp.local.-_daap._tcp.local.-_viziocast._tcp.local.-_wled._tcp.local.-_xbmc-jsonrpc-h._tcp.local.-_spotify-connect._tcp.local._827"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2407)
    run (zeroconf/__init__.py:1701)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 828 (idle): "Thread-48"
    send_events (pyhap/accessory_driver.py:483)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 829 (idle): "Thread-49"
    select (selectors.py:415)
    serve_forever (socketserver.py:232)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 831 (idle): "Thread-51"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 832 (idle): "Thread-52"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 6015 (idle): "Thread-81"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 25348 (idle): "Thread-170"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 164849 (idle): "stream_worker"
    _stream_worker_internal (homeassistant/components/stream/worker.py:205)
    stream_worker (homeassistant/components/stream/worker.py:48)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 259889 (idle): "Thread-624"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 260522 (idle): "Thread-625"
    recv (pyhap/hap_server.py:779)
    recv_into (pyhap/hap_server.py:758)
    readinto (socket.py:669)
    handle_one_request (http/server.py:395)
    handle (http/server.py:429)
    __init__ (socketserver.py:720)
    __init__ (pyhap/hap_server.py:164)
    finish_request (pyhap/hap_server.py:943)
    process_request_thread (socketserver.py:650)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
bash-5.0#
bdraco commented 3 years ago

Do you have something that is constantly gathering logbook data? There are at least 5 separate logbook api requests running in your py-spy.

bdraco commented 3 years ago

There is a lot going on here, but at least https://github.com/home-assistant/core/pull/40250 and https://github.com/home-assistant/core/pull/40272 should help performance of templates.

bdraco commented 3 years ago
Screen_Shot_2020-09-20_at_8_36_20_AM

It looks like the core issue is that MQTT is flooding home assistant with state changed events.

gieljnssns commented 3 years ago

Do you have something that is constantly gathering logbook data? There are at least 5 separate logbook api requests running in your py-spy.

Not that I know logbook.yaml

exclude:
  domains:
    - sun
  entities:
    - sensor.since_last_boot
    - sensor.date
    - sensor.cpu_use
    - sensor.ram_free
    - sensor.ram_use
    - sensor.time
    - sensor.time_date
    - sensor.pihole_ads_blocked_today
    - sensor.pihole_ads_percentage_blocked_today
    - sensor.pihole_dns_queries_today

logger.yaml

default: error

recorder.yaml

db_url: !secret mysql
exclude:
  domains:
    - camera
    - group
    - history_graph
    - scene
    - sun
    - weather
    - zone
  entities:
    - binary_sensor.bluetooth_scan
    - sensor.dark_sky_cloud_coverage_1
    - sensor.dark_sky_cloud_coverage_2
    - sensor.dark_sky_cloud_coverage_3
    - sensor.dark_sky_cloud_coverage_4
    - sensor.dark_sky_cloud_coverage_5
    - sensor.dark_sky_cloud_coverage_6
    - sensor.dark_sky_cloud_coverage_7
    - sensor.dark_sky_daily_summary
    - sensor.dark_sky_daytime_high_temperature
    - sensor.dark_sky_daytime_high_temperature_1
    - sensor.dark_sky_daytime_high_temperature_2
    - sensor.dark_sky_daytime_high_temperature_3
    - sensor.dark_sky_daytime_high_temperature_4
    - sensor.dark_sky_daytime_high_temperature_5
    - sensor.dark_sky_daytime_high_temperature_6
    - sensor.dark_sky_daytime_high_temperature_7
    - sensor.dark_sky_dew_point_1
    - sensor.dark_sky_dew_point_2
    - sensor.dark_sky_dew_point_3
    - sensor.dark_sky_dew_point_4
    - sensor.dark_sky_dew_point_5
    - sensor.dark_sky_dew_point_6
    - sensor.dark_sky_dew_point_7
    - sensor.dark_sky_hourly_summary
    - sensor.dark_sky_humidity_1
    - sensor.dark_sky_humidity_2
    - sensor.dark_sky_humidity_3
    - sensor.dark_sky_humidity_4
    - sensor.dark_sky_humidity_5
    - sensor.dark_sky_humidity_6
    - sensor.dark_sky_humidity_7
    - sensor.dark_sky_overnight_low_temperature
    - sensor.dark_sky_overnight_low_temperature_1
    - sensor.dark_sky_overnight_low_temperature_2
    - sensor.dark_sky_overnight_low_temperature_3
    - sensor.dark_sky_overnight_low_temperature_4
    - sensor.dark_sky_overnight_low_temperature_5
    - sensor.dark_sky_overnight_low_temperature_6
    - sensor.dark_sky_overnight_low_temperature_7
    - sensor.dark_sky_precip_1
    - sensor.dark_sky_precip_2
    - sensor.dark_sky_precip_3
    - sensor.dark_sky_precip_4
    - sensor.dark_sky_precip_5
    - sensor.dark_sky_precip_6
    - sensor.dark_sky_precip_7
    - sensor.dark_sky_precip_intensity_1
    - sensor.dark_sky_precip_intensity_2
    - sensor.dark_sky_precip_intensity_3
    - sensor.dark_sky_precip_intensity_4
    - sensor.dark_sky_precip_intensity_5
    - sensor.dark_sky_precip_intensity_6
    - sensor.dark_sky_precip_intensity_7
    - sensor.dark_sky_summary
    - sensor.dark_sky_wind_bearing_1
    - sensor.dark_sky_wind_bearing_2
    - sensor.dark_sky_wind_bearing_3
    - sensor.dark_sky_wind_bearing_4
    - sensor.dark_sky_wind_bearing_5
    - sensor.dark_sky_wind_bearing_6
    - sensor.dark_sky_wind_bearing_7
    - sensor.dark_sky_wind_speed_1
    - sensor.dark_sky_wind_speed_2
    - sensor.dark_sky_wind_speed_3
    - sensor.dark_sky_wind_speed_4
    - sensor.dark_sky_wind_speed_5
    - sensor.dark_sky_wind_speed_6
    - sensor.dark_sky_wind_speed_7
    - sensor.date
    - sensor.date_time
    - sensor.moon
    - sensor.next_rising
    - sensor.notification_hour
    - sensor.notification_minute
    - sensor.notification_time
    - sensor.notification_time_long
    - sensor.people_in_space
    - sensor.pws_weather_1n_metric
    - sensor.pws_weather_2d_metric
    - sensor.pws_weather_2n_metric
    - sensor.pws_weather_3d_metric
    - sensor.pws_weather_3n_metric
    - sensor.pws_weather_4d_metric
    - sensor.pws_weather_4n_metric
    - sensor.since_last_boot
    - sensor.time
    - sensor.time_date
    - sensor.cert_expiry
    - sensor.custom_card_tracker
    - sensor.custom_component_tracker
    - sensor.custom_python_script_tracker
    - sensor.dummy
    - sensor.dummy_badkamer
    - sensor.dummy_deze_maand
    - sensor.dummy_slaapkamer
    - sensor.dummy_slaapkamer_fien
    - sensor.dummy_slaapkamer_noor
    - sensor.dummy_vandaag
    - sensor.dummy_verluchting
    - sensor.dummy_warmtepomp
    - sensor.last_boot
    - sensor.moon
    - sensor.notification_hour
    - sensor.notification_minute
    - sensor.notification_time
    - sensor.notification_time_long
    - sensor.uptime_in_dagen
    - sensor.uptime_in_uren
    - sensor.zonne_energie_omvormers_gefilterd_lowpass

history.yaml

exclude:
  domains:
    - weblink
    - automation
    - updater
    - sun
    - group
  entities:
    - sensor.since_last_boot
    - sensor.date
    - sensor.pws_dewpoint_c
    - sensor.pws_dewpoint_string
    - sensor.pws_feelslike_c
    - sensor.pws_feelslike_string
    - sensor.pws_heat_index_c
    - sensor.pws_heat_index_string
    - sensor.pws_precip_today_metric
    - sensor.pws_precip_today_string
    - sensor.pws_pressure_mb
    - sensor.pws_relative_humidity
    - sensor.pws_solarradiation
    - sensor.pws_station_id
    - sensor.pws_uv
    - sensor.pws_visibility_km
    - sensor.pws_weather
    - sensor.pws_wind_string
    - sensor.ram_use
    - sensor.time
    - sensor.time_date
    - sensor.weer_morgen_condition
    - sensor.weer_morgen_temperature_max
    - sensor.weer_morgen_temperature_min
    - sensor.weer_overmorgen_condition
    - sensor.weer_overmorgen_temperature_max
    - sensor.weer_overmorgen_temperature_min
    - binary_sensor.den_hof_outdoor_animal
    # - binary_sensor.nummer_79_den_hof_outdoor_animal
    - binary_sensor.den_hof_outdoor_motion
    - binary_sensor.den_hof_outdoor_vehicle
    # - binary_sensor.nummer_79_den_hof_outdoor_vehicle
    - climate.netatmo_living_2
    - binary_sensor.binnen_warm
    - binary_sensor.buiten_fris_genoeg
    - binary_sensor.buiten_warm
    - sensor.badkamertemperatuur_mean
    - sensor.beweging_badkamer_count
    - sensor.beweging_slaapkamer_count
    - sensor.beweging_slaapkamer_fien_count
    - sensor.beweging_slaapkamer_noor_count
    - sensor.buienradar
    - sensor.bureau
    - sensor.fien
    - sensor.noor
    - sensor.opladers
    - sensor.slaapkamer
    - sensor.temperatuur_living_mean
    - sensor.verluchting
    - sensor.wc
    - zone.home

It looks like the core issue is that MQTT is flooding home assistant with state changed events.

I have Smappee configured via MQTT, Every second I get a realtime update

{
    "totalPower": 833,
    "totalReactivePower": 528,
    "totalExportEnergy": 0,
    "totalImportEnergy": -675983779,
    "monitorStatus": 0,
    "utcTimeStamp": 1600610515811,
    "channelPowers": [
        {
            "ctInput": 0,
            "power": 146,
            "exportEnergy": 0,
            "importEnergy": -2136093549,
            "phaseId": 1,
            "current": 11
        },
        {
            "ctInput": 1,
            "power": 569,
            "exportEnergy": 0,
            "importEnergy": -1238023984,
            "phaseId": 0,
            "current": 26
        },
        {
            "ctInput": 2,
            "power": 118,
            "exportEnergy": 0,
            "importEnergy": -1596829640,
            "phaseId": 2,
            "current": 6
        },
        {
            "ctInput": 3,
            "power": 1005,
            "exportEnergy": 11198469,
            "importEnergy": -1071029410,
            "phaseId": 1,
            "current": 43
        },
        {
            "ctInput": 4,
            "power": 2075,
            "exportEnergy": 12407578,
            "importEnergy": 1014673999,
            "phaseId": 0,
            "current": 88
        },
        {
            "ctInput": 5,
            "power": 1000,
            "exportEnergy": 675661,
            "importEnergy": -1210750333,
            "phaseId": 2,
            "current": 42
        }
    ],
    "voltages": [
        {
            "voltage": 236,
            "phaseId": 0
        }
    ]
}
bdraco commented 3 years ago

Please try changing the commit interval in recorder to 5 seconds and make a new py-spy recording

gieljnssns commented 3 years ago

This is a new https://www.dropbox.com/s/elvi42f589055gs/spy.svg?dl=0

bdraco commented 3 years ago

That looks better. Is there a way to turn down the Smappee updates to every 5 seconds as well? That should calm things down until we can get https://github.com/home-assistant/core/pull/40345 https://github.com/home-assistant/core/pull/40250 and https://github.com/home-assistant/core/pull/40272 out in 0.116

gieljnssns commented 3 years ago

Is there a way to turn down the Smappee updates to every 5 seconds as well?

I can't change this...

This is a screenshot of glances when I'm clicking on different entity's in a Lovelace view Schermafbeelding 2020-09-20 om 16 20 34

bdraco commented 3 years ago

The high cpu there is caused by showing the logbook part on the more info panel. You have so many state changes feeding into the database that its taking a long time to fetch all the events for the past 24 hours each time you click the popup.

cc @zsarnett

zsarnett commented 3 years ago

Maybe we should only show the last 12 hours or last x entries? Not sure what the best solution would be for this. I wonder how many others have this same issue

Mariusthvdb commented 3 years ago

Would be very nice to have the logbook on the More-info card optional in the first place

Secondly, please make that only show on entities included in logbook/history.

Right now it shows on all entities with the swirling icon and after a second or 5 stating nothing to show...

Only posting here since you are discussing optimizing this. Otherwise I would have added a new FR

bdraco commented 3 years ago

Maybe we should only show the last 12 hours or last x entries? Not sure what the best solution would be for this. I wonder how many others have this same issue

I'll work on the query on the backend. We may be able to make it faster by excluding all the state change events and then unioning them back in if it doesn't make the query too complex. We may have to add a time_type index

bdraco commented 3 years ago

Maybe we should only show the last 12 hours or last x entries? Not sure what the best solution would be for this. I wonder how many others have this same issue

@zsarnett No need to make any changes, I fixed the performance issue looking up single entities in https://github.com/home-assistant/core/pull/40075.

Loading my garage exit door lock status before 1.32s (1324ms) after 94ms

bdraco commented 3 years ago

I think the logbook access in the more info is the root cause of some of the other performance issues in 0.115. The problem should just go away after #40075

gieljnssns commented 3 years ago

I made a custom component for logger with your changes in #40075 This much better, when I'm clicking around in Lovelace, no more blowing fans This is a Py-Spy when these changes are applied https://www.dropbox.com/s/t5zqcq9dzl4v5nk/spy-40075.svg?dl=0 And this is a screenshot of glances 40075

Only my idle CPU use is still more then 0.114.x and I still have occasionally blowing fans

gieljnssns commented 3 years ago

I made another custom component from #40345 (I took the MQTT Component inn DEV and applied your changes) This is not so good, constantly blowing fans 2 Py-Spy's https://www.dropbox.com/s/u095q9djk0txmcr/spy-40075%2B40345.svg?dl=0 https://www.dropbox.com/s/yvcxkmrmrk1xo6f/spy-40075%2B40345_2.svg?dl=0 And a screenshot 40075+40345

gieljnssns commented 3 years ago

I don't know how to test #40250 and #40272 so for now I just leave my custom component for logbook applied

gieljnssns commented 3 years ago

This is another Py-Spy, immediately started when the fans of my NUC started to blow https://www.dropbox.com/s/iustxvojg8v07gn/spy-40075-blowing.svg?dl=0

This is one when the fans started to blow at around 1min 40sec https://www.dropbox.com/s/c3kq8lc9m30qu2s/spy-40075-blowing-fan-2.svg?dl=0

This are 2 Py-Spy's of 300sec

bdraco commented 3 years ago

Can you post your template entities (sensors) configurtion?

gieljnssns commented 3 years ago

The most of my template sensors are in sensors https://github.com/gieljnssns/My-Hassio-config/tree/master/config/devices/sensors/template_sensors And some are in packages https://github.com/gieljnssns/My-Hassio-config/tree/master/config/packages

In here are a lot of binary sensors binary_sensors https://github.com/gieljnssns/My-Hassio-config/tree/master/config/devices/binary_sensors

In here are some template switches switches https://github.com/gieljnssns/My-Hassio-config/tree/master/config/devices/switch

bdraco commented 3 years ago

Try disabling config/devices/sensors/template_sensors/unavailable.yaml and template_sensors/component_count.yaml

Also if you change the sensors in: template_sensors/component_count.yaml to {{ states.input_text | count }} instead of {{ states.input_text | list | length }} since they no longer have to generate a list and throw it away.

gieljnssns commented 3 years ago

I'm sorry, I forgot to tell config/devices/sensors/template_sensors/unavailable.yaml Is already disabled for 2 days, until I make an automation for it And {{ states.input_text | list | length }} is already converted to {{ states.input_text | count }}

And I also replaced all now() by as_local(states.sensor.time.last_changed)

Sorry for the confusion, my repo isn't updated yet