search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
69.79k stars 28.93k forks source link

homeassistant.components.recorder.core #121164

Open matomatusov opened 3 days ago

matomatusov commented 3 days ago

The problem

Hi, I don't know what could be the problem? There is free disk space.

Logger: homeassistant.components.recorder.core Source: components/recorder/core.py:395 integration: Recorder (documentation, issues) First occurred: 5:21:15 AM (1 occurrences) Last logged: 5:21:15 AM

The recorder backlog queue reached the maximum size of 72299 events; usually, the system is CPU bound, I/O bound, or the database is corrupt due to a disk problem; The recorder will stop recording events to avoid running out of memory

1 2

What version of Home Assistant Core has the issue?

core-2024.7.0

What was the last working version of Home Assistant Core?

core-2024.6.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

ohkaja commented 3 days ago

Looks equal to https://github.com/home-assistant/frontend/issues/21278 seems restarting HA twice will fix this.

image
matomatusov commented 3 days ago

@ohkaja The HA reset didn't help me. But thanks for the info.

home-assistant[bot] commented 3 days ago

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `recorder` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign recorder` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

recorder documentation recorder source (message by IssueLinks)

ohkaja commented 2 days ago

@matomatusov

Then there is a recorder error, if i see right your´s stopped working at 4 a.m. too. I had/ have no log entries.

ChristophCaina commented 2 days ago

this is probably a known issue which should be fixed in next months release... I just need to find the pr... -> #117263

For the time being, I've created an automation to reboot / restart HomeAssistant before the purge job will start. This seems to help (for now) - but without purging the Database, the size of it will just grow.

MarcelNoir commented 2 days ago

I'm running into the same problem with the recorder. It has just stopped working after I updated to 2024.7.0 this morning. The log is as follows:

Logger: homeassistant.core Quelle: core.py:1559 Erstmals aufgetreten: 12:26:13 (1807 Vorkommnisse) Zuletzt protokolliert: 12:33:17

Error running job: <Job listen HassJobType.Callback <function Recorder.async_initialize.._event_listener at 0x7f81de39fb00>> Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/core.py", line 1559, in async_fire_internal self._hass.async_run_hass_job(job, event) File "/usr/src/homeassistant/homeassistant/core.py", line 938, in async_run_hass_job hassjob.target(args) File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 330, in _event_listener if entity_filter(entity_id): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

Logger: homeassistant.components.recorder.core Quelle: components/recorder/core.py:911 Integration: Recorder (Dokumentation, Probleme) Erstmals aufgetreten: 12:26:33 (2 Vorkommnisse) Zuletzt protokolliert: 12:30:10

Error while processing event CompileMissingStatisticsTask() Error while processing event StatisticsTask(start=datetime.datetime(2024, 7, 4, 10, 25, tzinfo=datetime.timezone.utc), fire_events=True) Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 911, in _guarded_process_one_task_or_event_or_recover self._process_one_task_or_event_or_recover(task) File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 931, in _process_one_task_or_event_or_recover task.run(self) File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 178, in run if statistics.compile_missing_statistics(instance): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 644, in wrapper return job(instance, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 477, in compile_missing_statistics modified_statistic_ids = _compile_statistics( ^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 559, in _compile_statistics compiled: PlatformCompiledStatistics = platform_compile_statistics( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 400, in compile_statistics sensor_states = _get_sensor_states(hass) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 92, in _get_sensor_states and (not entity_filter or entity_filter(state.entity_id)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

Logger: homeassistant.components.recorder.util Quelle: components/recorder/util.py:137 Integration: Recorder (Dokumentation, Probleme) Erstmals aufgetreten: 12:26:33 (2 Vorkommnisse) Zuletzt protokolliert: 12:30:10

Error executing query Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 137, in session_scope yield session File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 477, in compile_missing_statistics modified_statistic_ids = _compile_statistics( ^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 559, in _compile_statistics compiled: PlatformCompiledStatistics = platform_compile_statistics( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 400, in compile_statistics sensor_states = _get_sensor_states(hass) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 92, in _get_sensor_states and (not entity_filter or entity_filter(state.entity_id)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

ohkaja commented 2 days ago

Yeah. Its reproducable via recorder.purge, which is also triggered 4:12 a.m..

andyblac commented 2 days ago

this is probably related to this change https://github.com/home-assistant/core/commit/9082dc2a799734f7dfa7eb7cbcb8e99210796b80#diff-909bb3e2ee175002975532b69a04af5a78fd77c6ca3259f3848b7c1e0678cba1R321-R324

andyblac commented 2 days ago

I'm running into the same problem with the recorder. It has just stopped working after I updated to 2024.7.0 this morning. The log is as follows:

Logger: homeassistant.core Quelle: core.py:1559 Erstmals aufgetreten: 12:26:13 (1807 Vorkommnisse) Zuletzt protokolliert: 12:33:17

Error running job: <Job listen HassJobType.Callback <function Recorder.async_initialize.._event_listener at 0x7f81de39fb00>> Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/core.py", line 1559, in async_fire_internal self._hass.async_run_hass_job(job, event) File "/usr/src/homeassistant/homeassistant/core.py", line 938, in async_run_hass_job hassjob.target(args) File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 330, in _event_listener if entity_filter(entity_id): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

Logger: homeassistant.components.recorder.core Quelle: components/recorder/core.py:911 Integration: Recorder (Dokumentation, Probleme) Erstmals aufgetreten: 12:26:33 (2 Vorkommnisse) Zuletzt protokolliert: 12:30:10

Error while processing event CompileMissingStatisticsTask() Error while processing event StatisticsTask(start=datetime.datetime(2024, 7, 4, 10, 25, tzinfo=datetime.timezone.utc), fire_events=True) Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 911, in _guarded_process_one_task_or_event_or_recover self._process_one_task_or_event_or_recover(task) File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 931, in _process_one_task_or_event_or_recover task.run(self) File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 178, in run if statistics.compile_missing_statistics(instance): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 644, in wrapper return job(instance, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 477, in compile_missing_statistics modified_statistic_ids = _compile_statistics( ^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 559, in _compile_statistics compiled: PlatformCompiledStatistics = platform_compile_statistics( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 400, in compile_statistics sensor_states = _get_sensor_states(hass) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 92, in _get_sensor_states and (not entity_filter or entity_filter(state.entity_id)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

Logger: homeassistant.components.recorder.util Quelle: components/recorder/util.py:137 Integration: Recorder (Dokumentation, Probleme) Erstmals aufgetreten: 12:26:33 (2 Vorkommnisse) Zuletzt protokolliert: 12:30:10

Error executing query Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 137, in session_scope yield session File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 477, in compile_missing_statistics modified_statistic_ids = _compile_statistics( ^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 559, in _compile_statistics compiled: PlatformCompiledStatistics = platform_compile_statistics( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 400, in compile_statistics sensor_states = _get_sensor_states(hass) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/sensor/recorder.py", line 92, in _get_sensor_states and (not entity_filter or entity_filter(state.entity_id)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/icloud3/support/recorder_prefilter.py", line 123, in entity_filter return recorder_entity_filter(entity_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not callable

it seem tis is an issue with the iCloudv3 integration and the latest HA 2024.07, fix is here https://github.com/gcobb321/icloud3_v3/commit/9d8769d850f47f47cbb110856382088c80a82a82

ohkaja commented 2 days ago

this is probably related to this change https://github.com/home-assistant/core/commit/9082dc2a799734f7dfa7eb7cbcb8e99210796b80#diff-909bb3e2ee175002975532b69a04af5a78fd77c6ca3259f3848b7c1e0678cba1R321-R324

Nope. I've an Entity-Filter Set (not empty) but all logger entities are affected.

parklife200 commented 2 days ago

Had the same issue and resolved by following https://github.com/gcobb321/icloud3_v3/issues/308

andyblac commented 2 days ago

recorder still broke though, just remove the icloudv3 error.

talormanda commented 2 days ago

SmartSelect_20240704_143553_Home Assistant

Same issue for me. 4 am it started and fixed after a full system reboot.

uandco commented 2 days ago

I was facing the same issue and the iCloud v3 fix mentioned above solved the issue (after restarting HA). Of course recorded data between then 2024.7 update and the iCloud fix is gone.

image
jurgen2005 commented 2 days ago

Same problem here two night in a row now, recorder only starts after a reboot...

luismiparra commented 2 days ago

Same problem for me,

In the mean time I just disable the auto purge

https://www.home-assistant.io/integrations/recorder/#configuration-variables

 recorder:
  auto_purge: false

EDIT: DO NOT FORGET TO DISABLE THIS WHEN ISSUE FIXED! (Failure to do so will condemn your home assistant to exceeding the disk at some point in time!)

ChristophCaina commented 1 day ago

In the mean time I just disable the auto purge

https://www.home-assistant.io/integrations/recorder/#configuration-variables

 recorder:
  auto_purge: false

but don't forget to disable this, once the issue has been fixed :D (same applies for 'auto-reboot')

jurgen2005 commented 1 day ago

No fix in 7.1? judging from the release notes..

drothenberger commented 1 day ago

No fix in 7.1? judging from the release notes..

It did not fix it for me. With 7.0 and 7.1, recorder.purge simply never finishes, although it does not produce any errors. I test this by manually calling the recorder.purge service, waiting for 30 minutes, and then restarting Home Assistant. With 6.4, this works fine. With 7.0 and 7.1, I see a message in the logs that a recorder session didn't finish before the restart.

I don't have any of the noted custom integrations installed, and I don't see any errors from recorder in my logs.

Techdisk commented 1 day ago

Putting in my two cents in that I do not have any of the noted custom integrations installed and am having the same issue, and can reproduce it by calling recorder.purge. The only custom integrations I have (aside from HACS itself) are the Dyson one and HASS.Agent. I do have a few other custom integrations that add to the frontend like cards but those shouldn't be touching recorder.

As a note, it looks like the automated purge is done at 00:00 UTC, which is why the recorder stops recording at, say, 4 AM for those in EST.

I have found that I can restart the recorder by simply calling recorder.disable then recorder.enable, though this is obviously not a fix.

EDIT 1: I'm dumb; 4 AM EST is not 00:00 UTC.

EDIT 2: After a bit more testing, disabling and enabling the recorder with the services does not wake it back up; it only looks that way because your graphs start visually updating after you re-enable, but if you simply refresh your browser the history disappears again. You need to restart HA to get everything back to normal.

ChristophCaina commented 1 day ago

if you can reproduce the issue by calling recorder.purge, then you can just disable auto_purge from your configuration.yaml for now.

recorder:
  auto_purge: false

this should fix your issue until a fix has been rolled out. But don't forget to remove the change when the issue has been fixed, because else, your DB will just grow and this will cause other issues at some point :D

CommanderROR9 commented 17 hours ago

So this is not really related to any specific integration? I don't think I have any of the "suspects" running on my system and it's just the database/recorder purge that is causing issues for me...

cop98 commented 14 hours ago

I also have the same problem a little bit on all the integrations. For example I also notice it on the History stats

bbkbarbar commented 3 hours ago

I have the same problem too for example with a lot of my template_sensors even if the template sensor takes data only from core entity (e.g. azimuth attribute of "sun.sun" entity)

Afraskai commented 2 hours ago

In my instance, the purge never comes to an end since 2024.7.0. CPU Load starting at 4:12h every night and never stops. If there are too many transfers in the queue, the data remains in the home-assistant_v2.db-wal. This could be the reason why the graphs don´t update. I saw this also in the protocol of HA.

My solution is to roll back to 2024.6.4 till the issue is fixed.

On the way to find this bug report I stumbled on this good article to shrink and optimize the DB - it doesn´t help for this issue, but helped me a lot understanding the behavior of sqlite HA-DB:

https://www.vanwerkhoven.org/blog/2024/reduce-home-assistant-database-size/

quentinmit commented 2 hours ago

I'm experiencing the exact same behavior on 2024.6.4, but I'm using Postgres and not SQLite. The recorder stops writing state updates until Home Assistant is restarted.

Interestingly, my monitoring shows that Home Assistant is still performing nearly as many database transactions even when it's not recording history. I see lots of queries of the form:

UPDATE states SET last_reported_ts=1720335619.368038 WHERE states.state_id = 175733005

Is this also something that is likely to be fixed by the changes here, or is this actually a separate bug? I believe this started when I updated from 2024.6.1 to 2024.6.4, but it might have also started when I went from 2024.5.5 to 2024.6.1.