search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
4.92k stars 971 forks source link

Homeassistant not reachable after update to 13.2 #3629

Open observer-october opened 5 days ago

observer-october commented 5 days ago

Describe the issue you are experiencing

I updated from 13.1 to 13.2. With 13.1 everything worked fine, the instance was reachable and so on. After doing the update to 13.2 the HomeAssistant isn't reachable via web interface, tested with ip:8123 and homeassistant.local:8123, and iOS app. In Firefox and Vivaldi there is an "ERR_CONNECTION_REFUSED" Error shown and on the iOS app the error is "NSURLErrorDoman -1004" The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Best regards, Christoph

What operating system image do you use?

rpi5-64 (Raspberry Pi 5 64-bit OS)

What version of Home Assistant Operating System is installed?

13.2 (should be)

Did the problem occur after upgrading the Operating System?

Yes

Hardware details

RaspPi 5 with 8Gb Ram

Steps to reproduce the issue

  1. Update System to 13.2 coming from 13.1

Anything in the Supervisor logs that might be useful for us?

No connection to get some logs

Anything in the Host logs that might be useful for us?

No connection to get some logs

System information

No response

Additional information

No response

thlac commented 5 days ago

I got the same issue with the Home Assistant Yellow.

I'm connected via serial but can't real manage the system in any way, getting into the HA cli doesn't work, most docker commands are extremely slow,

journalctl -u docker, prints the following every 30 seconds:

Oct 15 20:41:30 homeassistant systemd[1]: Dependency failed for Docker Application Container Engine. Oct 15 20:41:30 homeassistant systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.

Edit: Digging in some more, it seems that after the upgrade it doesn't load my nvme-drive which my hassos-data is contained on.

lsblk doesn't show it, and the kernel doesn't seem to print much of relevance, other than:

[ 4.304412] systemd[1]: Expecting device /dev/disk/by-label/hassos-boot... [ 4.328305] systemd[1]: Expecting device /dev/disk/by-label/hassos-data... [ 4.352341] systemd[1]: Expecting device /dev/disk/by-label/hassos-overlay...

pamatika commented 5 days ago

Same here. Raspi 4 4gb installed on microsd. After upgrade from OS 13.1 to 13.2. System did not boot... recovering..

pcmike commented 5 days ago

Same here, had to restore to a previous backup. EDIT to add I’m on HAOS using official VM on proxmox and this is the first time an HAOS upgrade has resulted in an unusable system.

AndersHogqvist commented 5 days ago

Same here. I have it installed on a Raspberry Pi 5 with an NVMe and did the update from my phone.

How you do a recovery when you can't access it? I have a backup from just before the update. Do I have to reinstall the whole thing and then recover?

karambol commented 5 days ago

Same here on odroid

julianjd-uk commented 5 days ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD.

Did a update back to 13.1 from the command line.

agners commented 5 days ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD.

Did a update back to 13.1 from the command line.

This is probably related to #3352 :thinking:

agners commented 5 days ago

The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Anything shown on the system console? Are you using an SD card?

If the system doesn't boot three times in a row, it should revert to the previous release. So you can try removing power twice again and see if things startup correctly from then onwards.

agners commented 5 days ago

I got the same issue with the Home Assistant Yellow.

I'm connected via serial but can't real manage the system in any way, getting into the HA cli doesn't work, most docker commands are extremely slow,

journalctl -u docker, prints the following every 30 seconds:

Oct 15 20:41:30 homeassistant systemd[1]: Dependency failed for Docker Application Container Engine. Oct 15 20:41:30 homeassistant systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.

Edit: Digging in some more, it seems that after the upgrade it doesn't load my nvme-drive which my hassos-data is contained on.

lsblk doesn't show it, and the kernel doesn't seem to print much of relevance, other than:

[ 4.304412] systemd[1]: Expecting device /dev/disk/by-label/hassos-boot... [ 4.328305] systemd[1]: Expecting device /dev/disk/by-label/hassos-data... [ 4.352341] systemd[1]: Expecting device /dev/disk/by-label/hassos-overlay...

Hm, I do run my production system on Yellow from NVMe as well (using data disk), and here 13.2 does boot correctly.

Maybe a power cycle helps? :thinking: If this continuous to be a problem, probably better to open a separate issue.

agners commented 5 days ago

Same here on odroid

What ODROID exactly? Can you also check the console on HDMI if something in particular is printed?

observer-october commented 5 days ago

The RaspberryPi itself is available via Ping and also shown in the router configuration. A restart of the host hasn't helped.

Anything shown on the system console? Are you using an SD card?

If the system doesn't boot three times in a row, it should revert to the previous release. So you can try removing power twice again and see if things startup correctly from then onwards.

Yes, HA is running on a microsd. Restarting multiple times didn't work. I will remove from server rack later and have a look at it via HDMI.

karambol commented 5 days ago

Same here on odroid

What ODROID exactly? Can you also check the console on HDMI if something in particular is printed?

odroid xu4. it boot and reboot in few minutes: https://dpaste.org/Xt10P

julianjd-uk commented 5 days ago

Mine restarted but it couldn't see my zigbee or USB sticks. RPi 4 8g booted from USB SSD. Did a update back to 13.1 from the command line.

This is probably related to #3352 🤔

Maybe it is, but I've had no problems until upgrading to 13.2. I have a SkyConnect and that was working fine, it's only used for Matter devices. My Sonoff stick and ASUS bluetooth stick couldn't be seen.

TekFan commented 4 days ago

Similar issue here. Context: Rpi5 booting from nvme. I had absolutely no problem with rc1 and rc2, so I confidently updated to 13.2 release and everything went south. HA became slow or even unresponsive. So connected to console and it was full of nvme I/O errors... Same thing after a power-cycle. Then via console, I was lucky enough to be able to do a "os boot-slot other" and this rebooted back to rc2. This Pi5 runs now flawlessly again on rc2.

So it seems something was broken between rc2 and release concerning nvme driver.

gclenaghan commented 4 days ago

I have a Green and since upgrading to 13.2 it is unresponsive to any requests sent to homeassistant.local or its ip address. Add-ons are also inaccessible. Interestingly, if you access it via cloudflare tunnel it looks like its running normal. I don't have it setup to access addons directly that way but I can see in the logs they're running like normal. Automations have also been running as normal as best I can tell.

I've rebooted it twice to no effect.

ykarulin commented 4 days ago

I have the same issue with Raspberry Pi 4 b.

I have also tried clear reinstall, it didn't help. Connecting via ssh and getting logs gave me this:

# ha core logs
2024/10/16 10:42:38 Start mDNS broadcast
2024/10/16 10:42:38 Publish http://192.168.1.105:8123 to _home-assistant._tcp
2024/10/16 10:42:38 Start webserver on http://0.0.0.0:8123
2024/10/16 10:48:22 Start mDNS broadcast
2024/10/16 10:48:22 Publish http://192.168.1.105:8123 to _home-assistant._tcp
2024/10/16 10:48:22 Start webserver on http://0.0.0.0:8123
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun home-assistant (no readiness notification)
s6-rc: info: service legacy-services successfully started
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 12, in <module>
    from .const import REQUIRED_PYTHON_VER, RESTART_EXIT_CODE, __version__
  File "/usr/src/homeassistant/homeassistant/const.py", line 16, in <module>
    from .util.event_type import EventType
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 17, in <module>
    from .dt import as_local, utcnow
  File "/usr/src/homeassistant/homeassistant/util/dt.py", line 14, in <module>
    import ciso8601
ImportError: Error loading shared library /usr/local/lib/python3.12/site-packages/ciso8601.cpython-312-aarch64-linux-musl.so: Exec format error
[10:51:27] INFO: Home Assistant Core finish process exit code 1
[10:51:27] INFO: Home Assistant Core service shutdown
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
Bonehead999 commented 4 days ago

If you type banner at the CLI it restores service to HA, however im not sure how long for.

David1940 commented 4 days ago

I also cannot reach HA after upgrading to Operating System 13.2 from 13.1. Core 2024.10.2 Supervisor 2024.10.2 Operating System13.1 (pre-upgrade) Frontend 20241002.3 Running on VirtualBox 7.0 on a windows 10 host. I reverted to a VM backup with OS 13.1 that I created before the upgrade.

AndersHogqvist commented 4 days ago

Well, turned out to be a false alarm in my case. Found my Raspberry Pi turned off when I came home from work and after turning it on again it seems to work fine

Axmexz commented 4 days ago

I'm experiencing the same issues shortly after upgrading.

My Home Assistant Yellow died, and I'm currently not home to check it in depth. Power cycling it via my switch by turning POE on and off didn't change anything.

I can't ping the device, nor is it sending out any traffic.

gclenaghan commented 4 days ago

I'm not sure what changed but sometime in the last two hours my Green became accessible locally again and everything seems normal. The uptime sensor shows no resets since I last checked when it wasn't even responding to pings.

I have a Green and since upgrading to 13.2 it is unresponsive to any requests sent to homeassistant.local or its ip address. Add-ons are also inaccessible. Interestingly, if you access it via cloudflare tunnel it looks like its running normal. I don't have it setup to access addons directly that way but I can see in the logs they're running like normal. Automations have also been running as normal as best I can tell.

I've rebooted it twice to no effect.

thlac commented 4 days ago

Hm, I do run my production system on Yellow from NVMe as well (using data disk), and here 13.2 does boot correctly.

Maybe a power cycle helps? 🤔 If this continuous to be a problem, probably better to open a separate issue.

At the point I wrote I had already power cycled it multiple times, that's why I got out the serial cable 😄

For whatever it's worth, I did a reinstall with the Yellow Installer that put me back to 13.1, it was able to work out my existing data on the NVMe disk and practically resume as nothing had happened. This time I was able to see with ha os info that the 13.2 install was marked as bad. Ran the update with ha os update an that update went through with no issue.

Still a bit surprising to me that HAOS lost the NVMe disk entirely 🤷

Toxo666 commented 4 days ago

same here. on proxmox vm after upgrade from 13.1 to 13.2 the system is not accessible after reboot.

ncodee commented 4 days ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

TekFan commented 4 days ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

typing "os boot-slot other" at the ha> prompt in console should put you back on previous version.

agners commented 4 days ago

For whatever it's worth, I did a reinstall with the Yellow Installer that put me back to 13.1, it was able to work out my existing data on the NVMe disk and practically resume as nothing had happened. This time I was able to see with ha os info that the 13.2 install was marked as bad. Ran the update with ha os update an that update went through with no issue.

Still a bit surprising to me that HAOS lost the NVMe disk entirely 🤷

That indeed is a bit curious, I wonder if the update didn't not complete or corrupted which lead to not all Linux kernel modules being loaded or something. Hard to tell. Glad a second update resolved your issue! :+1:

agners commented 4 days ago

odroid xu4. it boot and reboot in few minutes: https://dpaste.org/Xt10P

It seems it happens during cleanup operations. Either the system runs out of memory and crashes, or some storage issue, I'd guess. Is this running from a SD card?

agners commented 4 days ago

My instance keeps crashing on Proxmox VM after a while. It seems like some sort of kernal panic. Are there easier ways in reverting back, instead of restoring backup?

If you have the stack trace of the crash that would be interesting for further investigation. But this should be a separate issue, as your instance comes up at first but crashes later.

vans113 commented 3 days ago

I haven't crashed yet but my HA is running very slow and sluggish. Did another reboot and took ages.

tescophil commented 3 days ago

haven't been able to update since 12.4, all subsequent version, including the latest 13.2 causes the system to boot for about a minute, then restart