home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.73k stars 29.99k forks source link

Upgrading to 2024.5.0b0 fails and auto rollback to 2024.4.4 or boot-loop #116131

Closed Rockman18 closed 4 months ago

Rockman18 commented 5 months ago

The problem

I try to update from 2024.4.4 to 2024.5.0b0 but after the update the system and plugins start and i don't know why but restart and rollback to 2024.4.4

I can't find any relevant logs 😞

[EDIT] After 4-5 tries, I'm in a boot loop.

What version of Home Assistant Core has the issue?

2024.5.0b0

What was the last working version of Home Assistant Core?

2024.4.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

home-assistant-rollback.log

2024-04-24 22:34:20.034 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:20.035 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration ovh which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:20.035 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration portainer which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:20.036 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration webrtc which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:20.037 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration xiaomi_miot which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:20.037 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration dreame_vacuum which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-04-24 22:34:22.633 ERROR (MainThread) [homeassistant.components.hassio.handler] Client error on /addons/a0d7b954_portainer/changelog request 0, message='Attempt to decode JSON with unexpected mimetype: application/octet-stream', url=URL('http://172.30.32.2/addons/a0d7b954_portainer/changelog')
2024-04-24 22:34:22.633 WARNING (MainThread) [homeassistant.components.hassio.data] Could not fetch changelog for a0d7b954_portainer: 
2024-04-24 22:34:24.587 WARNING (ImportExecutor_0) [homeassistant.helpers.typing] HomeAssistantType was used from webrtc, this is a deprecated alias which will be removed in HA Core 2025.5. Use homeassistant.core.HomeAssistant instead, please report it to the author of the 'webrtc' custom integration
2024-04-24 22:34:24.589 WARNING (ImportExecutor_0) [homeassistant.helpers.typing] ServiceCallType was used from webrtc, this is a deprecated alias which will be removed in HA Core 2025.5. Use homeassistant.core.ServiceCall instead, please report it to the author of the 'webrtc' custom integration
2024-04-24 22:34:30.651 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues
2024-04-24 22:34:30.912 WARNING (MainThread) [homeassistant.components.freebox.router] Home access is not granted
2024-04-24 22:34:31.349 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'xiaomi_miot' accesses hass.helpers.service. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from service directly at custom_components/xiaomi_miot/__init__.py, line 575: hass.helpers.service.async_register_admin_service(, please create a bug report at https://github.com/al-one/hass-xiaomi-miot/issues
2024-04-24 22:34:32.194 WARNING (MainThread) [homeassistant.components.heos] 192.168.1.91 is not logged in to a HEOS account and will be unable to retrieve HEOS favorites: Use the 'heos.sign_in' service to sign-in to a HEOS account
2024-04-24 22:34:36.451 WARNING (MainThread) [custom_components.xiaomi_miot.sensor] Get xiaomi scene history for 178387150 999001001560 failed: {'code': 0, 'message': '成功', 'result': {'history': []}}
2024-04-24 22:34:46.113 WARNING (MainThread) [homeassistant.helpers.entity] Entity camera.dark_vador_map (<class 'custom_components.dreame_vacuum.camera.DreameVacuumCameraEntity'>) is using deprecated supported features values which will be removed in HA Core 2025.1. Instead it should use <CameraEntityFeature: 0>, please create a bug report at https://github.com/Tasshack/dreame-vacuum/issues and reference https://developers.home-assistant.io/blog/2023/12/28/support-feature-magic-numbers-deprecation

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

joostlek commented 5 months ago

There usually are logs in home-assistant.log.1 or log.fault, do you see anything like this?

Rockman18 commented 5 months ago

I've found home-assistant-rollback.log and attached it to my first message but not very usefull home-assistant.log.fault is 113MB... I don't know what is relevant here.

robercy commented 5 months ago

Same problem here!

milandzuris commented 5 months ago

same problem

robercy commented 5 months ago

Suddenly worked. To me, problem solved.

tomlut commented 5 months ago

Same issue here. Logs attached.

home-assistant.log.fault.txt

home-assistant.log.1.txt

bdraco commented 5 months ago

https://github.com/home-assistant/core/pull/116168 should fix tomlut's issue, but for everyone else who hasn't posted home-assistant.log.fault, its impossible to tell if that will help. You might have a different problem

milandzuris commented 5 months ago

116168 should fix tomlut's issue, but for everyone else who hasn't posted home-assistant.log.fault, its impossible to tell if that will help. You might have a different problem

I downgrade to 2024.4.4 and now have same problem from new Beta, Fault is Clear nothing strange in log.

milandzuris commented 5 months ago

image

SeanPM5 commented 5 months ago

Just to add another data point, was getting the crash / rollback with Odroid-N2 (Home Assistant Blue) on Home Assistant OS. After the fix in Supervisor update 2024.04.4 the beta upgrade went through successfully.

Attached log below home-assistant.log.fault.txt

bdraco commented 4 months ago

2024.5.0b1 is available that should fix the issue shown in tomlut and SeanPM5 logs

Rockman18 commented 4 months ago

Still rollbacking again and again at each update try. But this time my fault.log stays empty.

bdraco commented 4 months ago
  1. Install profiler integration https://my.home-assistant.io/redirect/config_flow_start/?domain=profiler
  2. Enable asyncio debug service as soon as possible after startup https://www.home-assistant.io/integrations/profiler/#service-profilerset_asyncio_debug
  3. Watch logs for RuntimeError: Non-thread-safe operation and long asyncio delays
  4. Download and post logs with full trace

If you can't get it to boot up (or it rolls back right away), you can use debugpy: in configuration.yaml but its much slower

bdraco commented 4 months ago

If that still doesn't work, disable all custom components, and try upgrading

Scags104 commented 4 months ago

Took me like 50 tries but disabling local-tuya and one more hard reboot fixed it for me. Not bootlooping anymore

Thanks @bdraco

bdraco commented 4 months ago

Still rollbacking again and again at each update try. But this time my fault.log stays empty.

Any change in b2?

Rockman18 commented 4 months ago

I made the update just before b2 release. Quiet the same as @Scags104, I had to deactivate all my custom integrations. The update is ok now, I'm on b1. But I have to check all my custom integrations, I have a lot of :

I will try the b2 update later this evening

Rockman18 commented 4 months ago

It seems that my dreame-vaccum integration is the root cause of my issue. It was the last integration I had not reactivated yet. I just tried to activate it again and my home assistant froze instantly.

bdraco commented 4 months ago

You might be able to find the bad code in the integration if you use upgrade with it off/removed, than turn on debug mode and restart with

homeassistant:
  debug: true

After restarting enable asyncio debug using the profiler service in the profiler integration https://my.home-assistant.io/redirect/config_flow_start/?domain=profiler https://www.home-assistant.io/integrations/profiler/#service-profilerset_asyncio_debug

Than setup the integration again

Watch the logs and report anything to the integration's GitHub

Than turn it back off until it's fixed and you can safely upgrade

bdraco commented 4 months ago

https://github.com/Tasshack/dreame-vacuum/releases/tag/v1.0.3

Also there were some recent fixes for crashes in that integration in case it's a simple case of running an old version

Rockman18 commented 4 months ago

https://github.com/Tasshack/dreame-vacuum/releases/tag/v1.0.3

Also there were some recent fixes for crashes in that integration in case it's a simple case of running an old version

I'm already on v1.0.3 but thanks. I will try to debug the integration code.

bdraco commented 4 months ago

If you find something, and it's cheap to block the bad operation I'll take a look before beta is over.

al-one commented 4 months ago

xiaomi_miot has fixed this issue in the latest version v0.7.18.

bdraco commented 4 months ago

Since the source has been located, I'm going to close this issue so others don't pile on here given the title is quite generic and could be caused by many different problems, and each case should get its own investigation in a fresh issue.

bdraco commented 4 months ago

For anyone reading this issue, if you have a similar problem:

  1. Install profiler integration https://my.home-assistant.io/redirect/config_flow_start/?domain=profiler
  2. Enable asyncio debug service as soon as possible after startup https://www.home-assistant.io/integrations/profiler/#service-profilerset_asyncio_debug
  3. Watch logs for RuntimeError: Non-thread-safe operation
  4. Download and post logs with full trace

If you can't get the UI to even come up instead add the following to configuration.yaml and restart:

debugpy:

For 2024.5.x and later Home Assistant debug mode can also be enabled in configuration.yaml

homeassistant:
  debug: true

Open a new issue with the results

SeppOswald commented 4 months ago

Just updated today from 2024.4.4 to 2024.5.1. and realized that after doing so ecoflow cloud integration did not work correctly anymore. Especially delta max data did not arrive at the dashboard correctly and automation did not work correctly anymore. Since this is part of a working setup of my solar power equipment, I did no further tests and analysis but looking first of all for a quick way to go back to 20244.4., which fortunately was possible from the terminal window by typing: ha core update --version 2024.4.4 From this everything was ok again! So please have a close look to that newest update!