search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
5.07k stars 992 forks source link

Homeassistant not reachable after update to 13.2 #3629

Open observer-october opened 1 month ago

observer-october commented 1 month ago

Describe the issue you are experiencing

I updated from 13.1 to 13.2. With 13.1 everything worked fine, the instance was reachable and so on. After doing the update to 13.2 the HomeAssistant isn't reachable via web interface, tested with ip:8123 and homeassistant.local:8123, and iOS app. In Firefox and Vivaldi there is an "ERR_CONNECTION_REFUSED" Error shown and on the iOS app the error is "NSURLErrorDoman -1004" The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Best regards, Christoph

What operating system image do you use?

rpi5-64 (Raspberry Pi 5 64-bit OS)

What version of Home Assistant Operating System is installed?

13.2 (should be)

Did the problem occur after upgrading the Operating System?

Yes

Hardware details

RaspPi 5 with 8Gb Ram

Steps to reproduce the issue

  1. Update System to 13.2 coming from 13.1

Anything in the Supervisor logs that might be useful for us?

No connection to get some logs

Anything in the Host logs that might be useful for us?

No connection to get some logs

System information

No response

Additional information

No response

thlac commented 1 month ago

I got the same issue with the Home Assistant Yellow.

I'm connected via serial but can't real manage the system in any way, getting into the HA cli doesn't work, most docker commands are extremely slow,

journalctl -u docker, prints the following every 30 seconds:

Oct 15 20:41:30 homeassistant systemd[1]: Dependency failed for Docker Application Container Engine. Oct 15 20:41:30 homeassistant systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.

Edit: Digging in some more, it seems that after the upgrade it doesn't load my nvme-drive which my hassos-data is contained on.

lsblk doesn't show it, and the kernel doesn't seem to print much of relevance, other than:

[ 4.304412] systemd[1]: Expecting device /dev/disk/by-label/hassos-boot... [ 4.328305] systemd[1]: Expecting device /dev/disk/by-label/hassos-data... [ 4.352341] systemd[1]: Expecting device /dev/disk/by-label/hassos-overlay...

pamatika commented 1 month ago

Same here. Raspi 4 4gb installed on microsd. After upgrade from OS 13.1 to 13.2. System did not boot... recovering..

pcmike commented 1 month ago

Same here, had to restore to a previous backup. EDIT to add I’m on HAOS using official VM on proxmox and this is the first time an HAOS upgrade has resulted in an unusable system.

AndersHogqvist commented 1 month ago

Same here. I have it installed on a Raspberry Pi 5 with an NVMe and did the update from my phone.

How you do a recovery when you can't access it? I have a backup from just before the update. Do I have to reinstall the whole thing and then recover?

karambol commented 1 month ago

Same here on odroid

julianjd-uk commented 1 month ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD.

Did a update back to 13.1 from the command line.

agners commented 1 month ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD.

Did a update back to 13.1 from the command line.

This is probably related to #3352 :thinking:

agners commented 1 month ago

The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Anything shown on the system console? Are you using an SD card?

If the system doesn't boot three times in a row, it should revert to the previous release. So you can try removing power twice again and see if things startup correctly from then onwards.

agners commented 1 month ago

I got the same issue with the Home Assistant Yellow.

I'm connected via serial but can't real manage the system in any way, getting into the HA cli doesn't work, most docker commands are extremely slow,

journalctl -u docker, prints the following every 30 seconds:

Oct 15 20:41:30 homeassistant systemd[1]: Dependency failed for Docker Application Container Engine. Oct 15 20:41:30 homeassistant systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.

Edit: Digging in some more, it seems that after the upgrade it doesn't load my nvme-drive which my hassos-data is contained on.

lsblk doesn't show it, and the kernel doesn't seem to print much of relevance, other than:

[ 4.304412] systemd[1]: Expecting device /dev/disk/by-label/hassos-boot... [ 4.328305] systemd[1]: Expecting device /dev/disk/by-label/hassos-data... [ 4.352341] systemd[1]: Expecting device /dev/disk/by-label/hassos-overlay...

Hm, I do run my production system on Yellow from NVMe as well (using data disk), and here 13.2 does boot correctly.

Maybe a power cycle helps? :thinking: If this continuous to be a problem, probably better to open a separate issue.

agners commented 1 month ago

Same here on odroid

What ODROID exactly? Can you also check the console on HDMI if something in particular is printed?

observer-october commented 1 month ago

The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Anything shown on the system console? Are you using an SD card?

If the system doesn't boot three times in a row, it should revert to the previous release. So you can try removing power twice again and see if things startup correctly from then onwards.

Yes, HA is running on a microsd. Restarting multiple times didn't work. I will remove from server rack later and have a look at it via HDMI.

karambol commented 1 month ago

Same here on odroid

What ODROID exactly? Can you also check the console on HDMI if something in particular is printed?

odroid xu4. it boot and reboot in few minutes: https://dpaste.org/Xt10P

julianjd-uk commented 1 month ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD. Did a update back to 13.1 from the command line.

This is probably related to #3352 🤔

Maybe it is, but I've had no problems until upgrading to 13.2. I have a SkyConnect and that was working fine, it's only used for Matter devices. My Sonoff stick and ASUS bluetooth stick couldn't be seen.

TekFan commented 1 month ago

Similar issue here. Context: Rpi5 booting from nvme. I had absolutely no problem with rc1 and rc2, so I confidently updated to 13.2 release and everything went south. HA became slow or even unresponsive. So connected to console and it was full of nvme I/O errors... Same thing after a power-cycle. Then via console, I was lucky enough to be able to do a "os boot-slot other" and this rebooted back to rc2. This Pi5 runs now flawlessly again on rc2.

So it seems something was broken between rc2 and release concerning nvme driver.

gclenaghan commented 1 month ago

I have a Green and since upgrading to 13.2 it is unresponsive to any requests sent to homeassistant.local or its ip address. Add-ons are also inaccessible. Interestingly, if you access it via cloudflare tunnel it looks like its running normal. I don't have it setup to access addons directly that way but I can see in the logs they're running like normal. Automations have also been running as normal as best I can tell.

I've rebooted it twice to no effect.

ykarulin commented 1 month ago

I have the same issue with Raspberry Pi 4 b.

I have also tried clear reinstall, it didn't help. Connecting via ssh and getting logs gave me this:

# ha core logs
2024/10/16 10:42:38 Start mDNS broadcast
2024/10/16 10:42:38 Publish http://192.168.1.105:8123 to _home-assistant._tcp
2024/10/16 10:42:38 Start webserver on http://0.0.0.0:8123
2024/10/16 10:48:22 Start mDNS broadcast
2024/10/16 10:48:22 Publish http://192.168.1.105:8123 to _home-assistant._tcp
2024/10/16 10:48:22 Start webserver on http://0.0.0.0:8123
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun home-assistant (no readiness notification)
s6-rc: info: service legacy-services successfully started
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 12, in <module>
    from .const import REQUIRED_PYTHON_VER, RESTART_EXIT_CODE, __version__
  File "/usr/src/homeassistant/homeassistant/const.py", line 16, in <module>
    from .util.event_type import EventType
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 17, in <module>
    from .dt import as_local, utcnow
  File "/usr/src/homeassistant/homeassistant/util/dt.py", line 14, in <module>
    import ciso8601
ImportError: Error loading shared library /usr/local/lib/python3.12/site-packages/ciso8601.cpython-312-aarch64-linux-musl.so: Exec format error
[10:51:27] INFO: Home Assistant Core finish process exit code 1
[10:51:27] INFO: Home Assistant Core service shutdown
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
Bonehead999 commented 1 month ago

If you type banner at the CLI it restores service to HA, however im not sure how long for.

David1940 commented 1 month ago

I also cannot reach HA after upgrading to Operating System 13.2 from 13.1. Core 2024.10.2 Supervisor 2024.10.2 Operating System13.1 (pre-upgrade) Frontend 20241002.3 Running on VirtualBox 7.0 on a windows 10 host. I reverted to a VM backup with OS 13.1 that I created before the upgrade.

AndersHogqvist commented 1 month ago

Well, turned out to be a false alarm in my case. Found my Raspberry Pi turned off when I came home from work and after turning it on again it seems to work fine

Axmexz commented 1 month ago

I'm experiencing the same issues shortly after upgrading.

My Home Assistant Yellow died, and I'm currently not home to check it in depth. Power cycling it via my switch by turning POE on and off didn't change anything.

I can't ping the device, nor is it sending out any traffic.

gclenaghan commented 1 month ago

I'm not sure what changed but sometime in the last two hours my Green became accessible locally again and everything seems normal. The uptime sensor shows no resets since I last checked when it wasn't even responding to pings.

I have a Green and since upgrading to 13.2 it is unresponsive to any requests sent to homeassistant.local or its ip address. Add-ons are also inaccessible. Interestingly, if you access it via cloudflare tunnel it looks like its running normal. I don't have it setup to access addons directly that way but I can see in the logs they're running like normal. Automations have also been running as normal as best I can tell.

I've rebooted it twice to no effect.

thlac commented 1 month ago

Hm, I do run my production system on Yellow from NVMe as well (using data disk), and here 13.2 does boot correctly.

Maybe a power cycle helps? 🤔 If this continuous to be a problem, probably better to open a separate issue.

At the point I wrote I had already power cycled it multiple times, that's why I got out the serial cable 😄

For whatever it's worth, I did a reinstall with the Yellow Installer that put me back to 13.1, it was able to work out my existing data on the NVMe disk and practically resume as nothing had happened. This time I was able to see with ha os info that the 13.2 install was marked as bad. Ran the update with ha os update an that update went through with no issue.

Still a bit surprising to me that HAOS lost the NVMe disk entirely 🤷

Toxo666 commented 1 month ago

same here. on proxmox vm after upgrade from 13.1 to 13.2 the system is not accessible after reboot.

ncodee commented 1 month ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

TekFan commented 1 month ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

typing "os boot-slot other" at the ha> prompt in console should put you back on previous version.

agners commented 1 month ago

For whatever it's worth, I did a reinstall with the Yellow Installer that put me back to 13.1, it was able to work out my existing data on the NVMe disk and practically resume as nothing had happened. This time I was able to see with ha os info that the 13.2 install was marked as bad. Ran the update with ha os update an that update went through with no issue.

Still a bit surprising to me that HAOS lost the NVMe disk entirely 🤷

That indeed is a bit curious, I wonder if the update didn't not complete or corrupted which lead to not all Linux kernel modules being loaded or something. Hard to tell. Glad a second update resolved your issue! :+1:

agners commented 1 month ago

odroid xu4. it boot and reboot in few minutes: https://dpaste.org/Xt10P

It seems it happens during cleanup operations. Either the system runs out of memory and crashes, or some storage issue, I'd guess. Is this running from a SD card?

agners commented 1 month ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

If you have the stack trace of the crash that would be interesting for further investigation. But this should be a separate issue, as your instance comes up at first but crashes later.

vans113 commented 1 month ago

I haven't crashed yet but my HA is running very slow and sluggish. Did another reboot and took ages.

tescophil commented 1 month ago

haven't been able to update since 12.4, all subsequent version, including the latest 13.2 causes the system to boot for about a minute, then restart

flecmart commented 1 month ago

Just wanted to leave a comment here for other users: for me everything went smooth as always. Running official Home Assistant OS on proxmox (x86).

I hesitated to update because there were so many reports of the update breaking the system even on official hardware. Today I took a look at https://analytics.home-assistant.io/ and realized the update went smooth for a lot of users - so for others wondering... backup and go 🚀

observer-october commented 1 month ago

For me the only solution was to reinstall and use the backup. Took the opportunity and replaced the microsd with a NVME SSD. At the moment everything is running smoothly again.

I will close the issue here, because at least for me it is solved.

kkstar44 commented 3 weeks ago

I'm totally lost, I got a reboot tonight because of power problem and now HA not accessible by anyway that I know, SSH not working also but I can ping it.... I absolutely don't know what to do :(

It's working on a raspi 4b.

willrnsantana commented 2 weeks ago

Same here. Odroid XU4 in boot loop after upgrading to 13.2 Did a dozen of fresh installs and restores but it always go back to boot loop Only fix so far (about 12hs stable now) was to downgrade back to 13.1

Some evidence if it helps: image

observer-october commented 2 weeks ago

Couple of days later and my problems are back.

Gonna rollback to 13.1 now.

semeion-net commented 2 weeks ago

Same here - reboot after update to 13.2 on RPi5 (with internal SD card, not USB storage) resulted in unreachable system with fan running high constantly. Power cycling two times reverted to 13.1, fixing the problem.