Core 2024.5.0 upgrade process stalling

RobertAmour commented 4 months ago

The problem

The upgrade from 2024.4.4 to 2024.5.0 is not progress and has been stuck on installing for the last 30+ mins. The pre-upgrade backup completes, I can see this on the backup NAS mount I use, but Supervisor logs imply backup has failed

What version of Home Assistant Core has the issue?

core-2024.5.0

What was the last working version of Home Assistant Core?

core-2024.4.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Core upgrade

Link to integration documentation on our website

No response

Diagnostics information

config_entry-hassio-b75162d12f154721a100cda05242e9d0.json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-05-01 21:10:07.897 WARNING (MainThread) [supervisor.auth] Unauthorized login for 'addons'
2024-05-01 21:10:31.990 WARNING (MainThread) [supervisor.homeassistant.module] Preparing backup of Home Assistant Core failed. Check HA Core logs.
2024-05-01 21:10:33.140 INFO (MainThread) [supervisor.homeassistant.module] Backing up Home Assistant Core config folder
2024-05-01 21:10:59.349 INFO (MainThread) [supervisor.homeassistant.module] Backup Home Assistant Core config folder done
2024-05-01 21:10:59.354 WARNING (MainThread) [supervisor.homeassistant.module] Error during Home Assistant Core backup. Check HA Core logs.
2024-05-01 21:10:59.356 INFO (MainThread) [supervisor.backups.manager] Backup 2e481e3f starting stage finishing_file
2024-05-01 21:11:01.088 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 2e481e3f completed
2024-05-01 21:11:01.088 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2024.5.0
2024-05-01 21:11:01.089 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2024.4.4 to ghcr.io/home-assistant/qemux86-64-homeassistant:2024.5.0
2024-05-01 21:11:01.089 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2024.5.0.
2024-05-01 21:17:17.988 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from a0d7b954_ssh
2024-05-01 21:17:17.991 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from a0d7b954_ssh
2024-05-01 21:17:17.997 INFO (MainThread) [supervisor.api.middleware.security] /host/info access from a0d7b954_ssh
2024-05-01 21:17:18.000 INFO (MainThread) [supervisor.api.middleware.security] /core/info access from a0d7b954_ssh

Additional information

No response

RobertAmour commented 4 months ago

Undertook a manual backup, and this works correctly. Attempting an upgrade with the Backup prior to upgrade checkbox deselected

bdraco commented 4 months ago

Please post home-assistant.log, home-assistant.log.1, and if it exists home-assistant.log.fault

adizanni commented 4 months ago

Hello all, not sure if it is related but I had a similar issue here. Upgraded to 2024.5.0 this morning with Backup checked. It seemed to have worked and one backup was created. When the system rebooted it was still showing the 2024.5.0 upgrade as if the upgrade failed. I retried the upgrade (this time without backup. Now the core appears to be in 2024.5.0 but home assistant is stuck in a infinite restart loop: after 5-6 minutes from the restart it is crashing and restarting. UI is available for 30 seconds during the warmup.

home assistant log is this for every restart iteration:

2024-05-02 10:00:22.652 WARNING (Recorder) [homeassistant.components.recorder.util] The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly
2024-05-02 10:00:22.696 WARNING (Recorder) [homeassistant.components.recorder.util] Ended unfinished session (id=195 from 2024-05-02 07:59:28.952487)
2024-05-02 10:00:22.896 ERROR (MainThread) [homeassistant.components.hassio.handler] Client error on /addons/0c969f44_sh3d-ha/changelog request 0, message='Attempt to decode JSON with unexpected mimetype: application/octet-stream', url=URL('http://172.30.32.2/addons/0c969f44_sh3d-ha/changelog')
2024-05-02 10:00:22.896 WARNING (MainThread) [homeassistant.components.hassio.data] Could not fetch changelog for 0c969f44_sh3d-ha:
2024-05-02 10:00:27.221 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues
2024-05-02 10:00:28.076 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'xiaomi_miot' accesses hass.helpers.service. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from service directly at custom_components/xiaomi_miot/__init__.py, line 575: hass.helpers.service.async_register_admin_service(, please create a bug report at https://github.com/al-one/hass-xiaomi-miot/issues
2024-05-02 10:00:31.805 WARNING (SyncWorker_0) [homeassistant.components.proxmoxve] Host 192.168.1.70 is not reachable
2024-05-02 10:00:33.743 WARNING (MainThread) [custom_components.xiaomi_miot.sensor] Get xiaomi scene history for 1763612936 268001035564 failed: {'code': 0, 'message': '成功', 'result': {'history': []}}
2024-05-02 10:00:33.779 WARNING (MainThread) [custom_components.xiaomi_miot.sensor] Get xiaomi scene history for 1763612936 268001024934 failed: {'code': 0, 'message': '成功', 'result': {'history': []}}
2024-05-02 10:00:34.612 ERROR (MainThread) [homeassistant.components.sensor] Platform esphome does not generate unique IDs. ID C0:49:EF:CC:11:98-sensor-shower_sb_current_duration is already used by sensor.masterbedroom1_1_shower_sb_current_duration - ignoring sensor.masterbedroom1-1_shower_sb_current_duration
2024-05-02 10:00:34.704 WARNING (MainThread) [homeassistant.components.sensor] Entity sensor.masterbedroom1_1_terrace_sound_level (<class 'homeassistant.components.esphome.sensor.EsphomeSensor'>) is using native unit of measurement 'dBA' which is not a valid unit for the device class ('signal_strength') it is using; expected one of ['dB', 'dBm']; Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+esphome%22
2024-05-02 10:00:35.075 WARNING (SyncWorker_5) [homeassistant.helpers.frame] Detected that custom integration 'hass_agent' calls async_create_task from a thread at custom_components/hass_agent/__init__.py, line 147: hass.async_create_task(handle_apis_changed(hass, entry, apis)), please report it to the author of the 'hass_agent' custom integration
2024-05-02 10:00:35.355 ERROR (SyncWorker_5) [homeassistant.util.logging] Exception in updated when handling msg on 'hass.agent/devices/DESKTOP-MQJFI7E': '{"serial_number":"d31c6de5-e0af-46f3-b21e-aaa4320962e5","device":{"identifiers":"hass.agent-DESKTOP-MQJFI7E","manufacturer":"LAB02 Research","model":"Microsoft Windows NT 10.0.22631.0","name":"DESKTOP-MQJFI7E","sw_version":"2022.14.0"},"apis":{"notifications":true,"media_player":true}}'
Traceback (most recent call last):
  File "/config/custom_components/hass_agent/__init__.py", line 147, in updated
    hass.async_create_task(handle_apis_changed(hass, entry, apis))
  File "/usr/src/homeassistant/homeassistant/core.py", line 816, in async_create_task
    self.verify_event_loop_thread("async_create_task")
  File "/usr/src/homeassistant/homeassistant/core.py", line 440, in verify_event_loop_thread
    frame.report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 162, in report
    _report_integration(what, integration_frame, level, error_if_integration)
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 203, in _report_integration
    raise RuntimeError(
RuntimeError: Detected that custom integration 'hass_agent' calls async_create_task from a thread at custom_components/hass_agent/__init__.py, line 147: hass.async_create_task(handle_apis_changed(hass, entry, apis)). Please report it to the author of the 'hass_agent' custom integration.

2024-05-02 10:00:35.355 WARNING (SyncWorker_5) [py.warnings] /usr/src/homeassistant/homeassistant/util/logging.py:120: RuntimeWarning: coroutine 'handle_apis_changed' was never awaited
  log_exception(format_err, *args)

2024-05-02 10:00:44.069 WARNING (MainThread) [homeassistant.helpers.entity] Update of number.c1c_e90079676_detection_sensitivity is taking over 10 seconds
2024-05-02 10:00:44.070 WARNING (MainThread) [homeassistant.helpers.entity] Update of number.c1c_e90078883_detection_sensitivity is taking over 10 seconds
2024-05-02 10:00:44.070 WARNING (MainThread) [homeassistant.helpers.entity] Update of number.c1c_e18888418_detection_sensitivity is taking over 10 seconds

as the system is restarting abruptly it is giving an issue about the database SQL Lite. I'm stuck as I cannot even apply a backup and I do not know how to restart in fail safe mode.

bdraco commented 4 months ago

RuntimeError: Detected that custom integration 'hass_agent' calls async_create_task from a thread at custom_components/hass_agent/init.py, line 147: hass.async_create_task(handle_apis_changed(hass, entry, apis)). Please report it to the author of the 'hass_agent' custom integration.

Please report the thread safety problem to the custom component

bdraco commented 4 months ago

Please see the guide in the forums

https://community.home-assistant.io/t/tracking-down-instability-issues-caused-by-integrations/724441

adizanni commented 4 months ago

RuntimeError: Detected that custom integration 'hass_agent' calls async_create_task from a thread at custom_components/hass_agent/init.py, line 147: hass.async_create_task(handle_apis_changed(hass, entry, apis)). Please report it to the author of the 'hass_agent' custom integration.

Please report the thread safety problem to the custom component

Hello @bdraco, first of all thanks for the explanation and the link (very useful for future reference). In fact it was not the HASS.Agent custom that caused the stalling, it is the Dreame Vacuum for which I cannot find any trace in the log. Probably it is another issue caused by the same root cause (wrong management of thread-safety). By disabling the Dreame Vacuum integration I was able to restart. Hass.Agent is still enabled but there are no entities so it is not causing any harm....

RobertAmour commented 4 months ago

Sorry was busy at work yesterday. It actually seems the upgrade eventually completed, but I can only assume took multiple hours. Not sure what happened. Logs attached Archive 3.zip

bdraco commented 4 months ago

It looks like you have at least one custom integration doing non-thread-safe operations:

RuntimeError: Detected that custom integration 'bosch' calls async_create_task from a thread at custom_components/bosch/__init__.py, line 293: self.hass.async_create_task(self.recording_sensors_update()). Please report it to the author of the 'bosch' custom integration.

If you still have stability issues, please see: https://community.home-assistant.io/t/2024-5-tracking-down-instability-issues-caused-by-integrations/724441/1

RobertAmour commented 4 months ago

Thanks @bdraco I'll raise it with the integration maintainer

issue-triage-workflows[bot] commented 1 month ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core