Zunhammer
commented
1 month ago Same for me
Closed talormanda closed 1 month ago
Zunhammer
commented
1 month ago Same for me
bsegers73
commented
1 month ago same for me. Updated to 2024.8 and it went even worse, so I used the backup to get back to 2024.7. Database is 11Gb and I'm trying to purge it to get it much smaller. After the normal purge at 4.11AM the logbook is not writing anymore. The only thing what worked is restart HA. After update to 2024.8 also the restart didn't work anymore, so get back to 2024.7 I tried also to purge manually but that also hangs the logbook and have to restart.
Zunhammer
commented
1 month ago A restart temporarily fixes it for me until the next day still in 2024.8. So I stay on 2024.8 but disabled auto pruge again until fixed.
home-assistant[bot]
commented
1 month ago Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
recorder documentation recorder source (message by IssueLinks)
bdraco
commented
1 month ago The safest way is to do a backup, than extract all the home-assistant_v2.db* files and send them in a compressed zip or tarball. I can put them into a test instance and than see if I can replicate the purge problem, or if a prior migration was unsuccessful, find the reason why.
You can share to my dropbox as nick@koston.org or gdrive as bdraco@gmail.com
talormanda
commented
1 month ago The safest way is to do a backup, than extract all the
home-assistant_v2.db*files and send them in a compressed zip or tarball. I can put them into a test instance and than see if I can replicate the purge problem, or if a prior migration was unsuccessful, find the reason why.You can share to my dropbox as nick@koston.org or gdrive as bdraco@gmail.com
Will get this ready and reply soon. Is there any compromise to my account(s) / security by handing this over?
bdraco
commented
1 month ago Unless you are storing passwords in the state machine (HA goes out of the way to prevent this in code reviews), the most worrisome thing in the database is likely GPS coordinates (likely the same as every app on your phone with location permissions)
talormanda
commented
1 month ago Unless you are storing passwords in the state machine (HA goes out of the way to prevent this in code reviews), the most worrisome thing in the database is likely GPS coordinates (likely the same as every app on your phone with location permissions)
The file has been uploaded and shared.
bdraco
commented
1 month ago I'll take a look after lunch. I'm working on getting a new aiohttp release done right now.
bdraco
commented
1 month ago It looks like it attempted to rebuild the table in database 4x and failed every time. The foreign key FOREIGN KEY(event_id) REFERENCES events (event_id) ON DELETE CASCADE, that causes the issue remains.
Before I start trying to figure out why the table rebuild failed, can you confirm how much free disk space you have?
talormanda
commented
1 month ago It looks like it attempted to rebuild the table in database 4x and failed every time. The foreign key
FOREIGN KEY(event_id) REFERENCES events (event_id) ON DELETE CASCADE,that causes the issue remains.Before I start trying to figure out why the table rebuild failed, can you confirm how much free disk space you have?
bdraco
commented
1 month ago 17G home-assistant_v2.db
https://www.home-assistant.io/integrations/recorder/#disk-space-requirements
Its likely unable to rebuild the table because its running out of disk space. To do a table rebuild, it has to create a temporary copy, and than replace the original table.
talormanda
commented
1 month ago I can try to move everything to another device where I have more storage allocated. Keep in mind before all this happened though, my DB was much smaller before. It only got this large because of the issue.
With that said, I can clone my VM and move it to another system where I can provide it more storage. Would you be able to tell me which recorder purge settings I need to run?
bdraco
commented
1 month ago I'll load it up after lunch and make sure it does try to rebuild it again. You should have an error every time you startup in the log that it fails to migrate. I take it you don't so there might be something else wrong as well.
bdraco
commented
1 month ago There was another change in #122069 which made the change I made in #121938 ineffective so it is not trying to migrate it again.
I'll need to work out a solution.
giacomo892
commented
1 month ago @bdraco sent an email with my db too which is affected by the issue. thanks
bdraco
commented
1 month ago @bdraco sent an email with my db too which is affected by the issue. thanks
Good news is its the same issue.
I'm working on a fix in https://github.com/home-assistant/core/pull/123388
bsegers73
commented
1 month ago My core-log gives the following error: Could it be the same problem? Database is 11Gb and 41Gb free.
2024-08-08 19:14:08.298 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-08-08 19:14:08.299 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/__init__.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 233, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-08-08 19:20:14.596 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-08-08 19:20:14.596 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/__init__.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 233, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-08-08 19:20:14.821 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-08-08 19:20:14.821 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/__init__.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 233, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-08-08 19:23:12.658 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2024-08-08 19:23:12.658 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
2024-08-08 19:23:12.659 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
2024-08-08 19:23:12.743 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
2024-08-08 19:23:12.743 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
2024-08-08 19:23:12.744 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system
2024-08-08 19:23:12.744 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
2024-08-08 19:23:12.745 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
2024-08-08 19:23:12.745 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
2024-08-08 19:23:12.745 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
2024-08-08 19:23:12.745 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
2024-08-08 19:23:12.747 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
2024-08-08 19:23:12.823 INFO (MainThread) [supervisor.resolution.check] System checks complete
2024-08-08 19:23:12.824 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
2024-08-08 19:23:13.631 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-08-08 19:23:13.632 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-08-08 19:23:13.632 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-08-08 19:25:45.659 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
2024-08-08 19:25:46.104 INFO (MainThread) [supervisor.jobs] 'MountManager.reload' blocked from execution, mounting not supported on system
2024-08-08 19:25:54.104 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-08-08 19:25:55.316 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/brenner-tobias/ha-addons repository
2024-08-08 19:25:55.405 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/egguy/grott-home-assistant-add-on repository
2024-08-08 19:25:55.420 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/jehy/hass-addons repository
2024-08-08 19:25:55.437 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/music-assistant/home-assistant-addon repository
2024-08-08 19:25:55.476 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt repository
2024-08-08 19:25:55.507 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/sabeechen/hassio-google-drive-backup repository
2024-08-08 19:25:55.570 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
2024-08-08 19:25:55.590 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
2024-08-08 19:25:55.637 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
2024-08-08 19:26:03.635 INFO (MainThread) [supervisor.store] Loading add-ons from store: 89 all - 0 new - 0 remove
2024-08-08 19:26:03.635 INFO (MainThread) [supervisor.store] Loading add-ons from store: 89 all - 0 new - 0 remove
2024-08-08 19:26:47.825 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-08-08 19:26:47.825 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/__init__.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 233, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-08-08 19:26:48.214 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-08-08 19:26:48.214 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/__init__.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 233, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket availableMy core-log gives the following error: Could it be the same problem?
Might be a different problem. Its not really possible to know without examining the database.
ivanfmartinez
commented
1 month ago @bdraco my database is very large now, more than 46GB (configured for 10 days), it does not lock the recorder, but apparently it does not pack. I still trying to understand.
I don´t know if my database fits in this case about failed migrations.
I have found that states_attributes is very large, with lot of data that are not used, maybe this big size I and others are having is related to this table contents.
I found tha recorder have a way to remove attributes, https://github.com/home-assistant/core/blob/c3ab72a1f9753070f8258695ee4caa28b21b03da/homeassistant/components/recorder/db_schema.py#L588 but apparently its only available for integrations, not for user configuration. Making this configurable is on the roadmap ?
Sn4k3E4t3r
commented
1 month ago Same problem. From 2024.7.2 on i have no reporter anymore. So i set the Value "recorder: auto_purge: false" i have an 40GB Database....
bdraco
commented
1 month ago If you have the issue, please share your database using the instructions above https://github.com/home-assistant/core/issues/123348#issuecomment-2276106936
If you don't want to share your database, please don't post me too comments as they notify everyone on this issue. Instead give the issue a 👍 in the opening text above.
bsegers73
commented
1 month ago The safest way is to do a backup, than extract all the
home-assistant_v2.db*files and send them in a compressed zip or tarball. I can put them into a test instance and than see if I can replicate the purge problem, or if a prior migration was unsuccessful, find the reason why.You can share to my dropbox as nick@koston.org or gdrive as bdraco@gmail.com
See the email. Curious what it could be.
bdraco
commented
1 month ago The safest way is to do a backup, than extract all the
home-assistant_v2.db*files and send them in a compressed zip or tarball. I can put them into a test instance and than see if I can replicate the purge problem, or if a prior migration was unsuccessful, find the reason why. You can share to my dropbox as nick@koston.org or gdrive as bdraco@gmail.comSee the email. Curious what it could be.
Good news. Its also the same issue
FOREIGN KEY(event_id) REFERENCES events (event_id) ON DELETE CASCADE,
https://github.com/home-assistant/core/pull/123388 is ready for review now so hopefully we can get it sorted in 2024.8.1
be sure to check https://www.home-assistant.io/integrations/recorder/#disk-space-requirements before updating
talormanda
commented
1 month ago How can we tell when this will be pushed to the next update?
bdraco
commented
1 month ago You can watch the milestone https://github.com/home-assistant/core/milestone/685
The date may change but its currently scheduled for today
bdraco
commented
1 month ago Please be sure your system meets the disk space requirements before updating to 2024.8.1
https://www.home-assistant.io/integrations/recorder/#disk-space-requirements
As there are multiple issue threads going, and your comments will likely be missed otherwise, please continue at https://community.home-assistant.io/t/psa-2024-7-recorder-problems/746428
The problem
My recorder has been locking up ever since the updates in 2024.7. I am now on 2024.8.0, and the issue still persists. Every night at 4:11 AM, it will lock up. Entities no longer get any history saved, and the
home-assistant_v2.dbfile no longer updates. If I turn off purging, or setpurge_keep_daysto something really far out, the issue is avoided temporarily. It is 4:39 AM as I type this, and my DB file has not updated since 4:11 AM:Nothing in this post fixed my issue: https://community.home-assistant.io/t/psa-2024-7-recorder-problems/746428/155
I did not have any of the mentioned integrations installed like others had.
I am adding 20 minutes worth of logs from
home-assistant.logat [4:00 AM to 4:20 AM].What version of Home Assistant Core has the issue?
2024.8.0
What was the last working version of Home Assistant Core?
2024.6
What type of installation are you running?
Home Assistant OS
Integration causing the issue
No response
Link to integration documentation on our website
No response
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
I am willing to provide my DB (privately) to someone who can look into / fix the issue, but it is rather large.