search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
4.79k stars 959 forks source link

Update 6.1 - 6.2 makes RPi4B unbootable #1490

Closed done7k closed 3 years ago

done7k commented 3 years ago

Hardware Environment

Home Assistant OS release:

Supervisor logs:

Journal logs:

Kernel logs:

Description of problem:

agners commented 3 years ago

Which version have you been using before? What type of USB 3.0 M.2 adapter are you using?

6.2 comes with a new firmware, but if 6.1 (with the old firmware) works and 6.2 (with the new firmware) reproducible doesn't work, then the best corse of action is to report the problem upstream with the Raspberry Pi firmware developers.

done7k commented 3 years ago

yes, it was on 6.1 before. then backup'ed and switched from SD card to USB3 SSD disk (Inteso M.2 SSD TOP SATA III 128Gb, in Gembird - EE2280-U3C-01- enclosure w/o external power supply). Also, I was experimenting today with images: Tried a fresh OS 6.2 (32bit) installation - same result - no boot/black screen tried a fresh OS 6.1 (32bit) installation - fine! then update to 6.2 from console - no boot /black screen; tried a fresh OS 6.2 (64bit) installation - it started fine but did not restore all add-ons… I replaced to 6.1 32bit

GiveMeTheBits commented 3 years ago

I also experienced this. I am still restoring because it would not apply my snapshots correctly. That is a separate issue, but I'm bummed out.

Raspberry Pi [4b] Updated from version 6.1 installed on a Samsung_Portable_SSD_T5 everything else was up to date as of yesterday except OS.

inverse commented 3 years ago

I previously had issues going from 5.12 -> 5.13 in a similar setup but going to 6.1 was stable. Will hold off the upgrade to 6.2 but following this.

agners commented 3 years ago

@GiveMeTheBits are you using 32-bit as well?

mstoffel-sag commented 3 years ago

Hi, I‘m not able to update from 6.1 to 6.2. might that be related? I’m also using an ssd:

21-08-05 06:30:21 ERROR (MainThread) [supervisor.hassos] Home Assistant Operating System update failed with: Installation error: Failed updating slot rootfs.1: Opening output device /dev/disk/by-partlabel/hassos-system1 failed: Device or resource busy

agners commented 3 years ago

@mstoffel-sag Do you have a SD card or something with an old HAOS installation still in that device? Does that still occures after rebooting?

GauthierCrp commented 3 years ago

Hello,

Same problem for me, after updating from 6.1 to 6.2, my raspberry pi 4 is unable to boot. I'm using a SSD. Then I restored from 6.1 on USB stick and try a new update... same result. 6.2 seems broken on RPI4 with USB boot. Should try with SD card.

I'm waiting for bug fix to update now.

Edit : don't know what this means (sadly not my domain...), but when I plug my usb stick to flash a new image, Windows finds many partitions on the device. Like 2 boot partitions of 24mo, 2 partitions of 256mo, one of 8mo, one of 96mo and then the leaving space with 27go. Too much to be normal isn't it ?

GiveMeTheBits commented 3 years ago

@GiveMeTheBits are you using 32-bit as well?

I was on 64-bit.

mstoffel-sag commented 3 years ago

@GauthierCrp I have no sd card (mount only shows sda mountpoints) but its weird. The update worked for me now. Looks like /dev/root at 100% is normal. What I changed though was: I removed a change that I did for mounting external media using udev. Got that from here: https://gist.github.com/eklex/c5fac345de5be9d9bc420510617c86b5

After I deleted the rule and did a reboot the update worked flawlessly.

Kind Regards,

Marco

agners commented 3 years ago

@mstoffel-sag thanks for the update. I think your issue is unrelated to what others are seeing here, as they have troubles booting after updating. 100% /dev/root is normal, we use a read-only squashfs, which basically is a minimal image for the whole root file system.

That said, the fact that your update succeeded now with USB SSD and others not shows that only some USB SSD installation are affected. Another round of detective work :cry:

@GauthierCrp 8 partitions is normal. We use that to separate kernel from the root file system, and to have two installations (A/B system, basically a fallback if booting one fails).

The booting problems are most likely caused by the upgrade to the lastest Raspberry Pi firmware. Unfortunately, its not the first time that previously working USB SSD installation got broken by newer Raspberry Pi firmwares.

To rule out any other problems, can you try the following: Flash 6.1, copy start.elf and start.dat from the first FAT partition (you might have to go to Disk Management or something to assign a letter/mount the partition). Connect the USB SSD back to the Rpi and update to 6.2. Verify boot is broken still. Connect the USB SSD to your PC again and replace start.elf and start.dat with the version from the 6.1 installation. Connect back to the Raspberry Pi and see if that makes 6.2 boot.

agners commented 3 years ago

As always: Not reproducible here :disappointed:

Using Raspberry Pi 4 Model B on a Samsung T1 series (MU-P8250B) attached via USB 3.0. Installed 64-bit HAOS 6.1, upgraded to 6.2 and rebooted successfully.

Bootloader: d6d82cf9 (Feb 16 2021) (disconnect all boot media, this should display the Raspberry Pi bootloader screen)

my_photo-74

mstoffel-sag commented 3 years ago

Thanks Stefan,

but since this is my "production" system I'm happy that it works.... hope you'll understand that..

agners commented 3 years ago

@mstoffel-sag sure. The firmware test was more meant for others having the issue.

done7k commented 3 years ago

@agners , Have not you tried 32bits image? or you believe it should not matter? As I mentioned with 64 bits I got 6.2 booted once but after I restored my Hassio configuration it did not restored all my add-ons. And when I started restoring them one by one it did not allowed me saying the system is in FREEZE state.. Thus I decided to move back on the 32bits. 6.1... not sure if it is correct to migrate from 6.1 32bit to 6.2 64bit ..

agners commented 3 years ago

I did not test 32-bit, I'll do that next week.

it did not restored all my add-ons.

Afaik, that should work. Which Add-On did not work? 64-bit is recommended now, so moving to 64-bit would be a good thing :smile:

done7k commented 3 years ago

@agners just have checked my board's bootloader. I flashed a new SD with fresh Raspberian; booted; it did something and rebooted automatically; then I removed SD and rebooted again. My bootloader dated by April 29 20210806_211533

done7k commented 3 years ago

Afaik, that should work. Which Add-On did not work? 64-bit is recommended now, so moving to 64-bit would be a good thing smile

Adguard_home; AppDaemon_4; Grocy; Zigbee2mqtt .. and some other ones but not very important. For instance I tried to restore AppDaemon and Adguard and got 'System in freeze state' msg.

GiveMeTheBits commented 3 years ago

@agners , Have not you tried 32bits image? or you believe it should not matter? As I mentioned with 64 bits I got 6.2 booted once but after I restored my Hassio configuration it did not restored all my add-ons. And when I started restoring them one by one it did not allowed me saying the system is in FREEZE state.. Thus I decided to move back on the 32bits. 6.1... not sure if it is correct to migrate from 6.1 32bit to 6.2 64bit ..

This is my experience as well. I assumed it was my Snapshots were incomplete or corrupt, but maybe it is a problem in the update? I was running 6.1 64-bit, updated to 6.2 and it died. wiped my SSD, installed fresh from RPI imager. I unfortunately did not check which OS version it installed, but I think it would pull the current one from github??? At this point, my snapshot restores were missing add-ons and I went back to my most recent full snapshot from March and tried to reapply the partial from July 30th, but I also got freeze state errors and issues with it actually applying anything. I wound up extracting the tarball and just copying my config folder back into my running config. :(

DonnyDepp commented 3 years ago

Same here... Since the latest update homeassistant became totally unresponsive and is always getting network timeouts. The network port led is blinking and going out in a loop.

agners commented 3 years ago

Finally came around testing with 32-Bit. I can reproduce the issue. Enabling UART via enable_uart=1 in config.txt shows that U-Boot hangs:

U-Boot 2021.04 (Jul 28 2021 - 22:21:10 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found
USB is stopped. Please issue 'usb start' first.
starting USB...
Bus xhci_pci: Host not halted after 16000 microseconds.
probe failed, error -16
No working controllers found

It seems that enabling compressed kernel support (via https://github.com/home-assistant/operating-system/pull/1418) changed U-Boot slightly, which changed timing and causes this issue. Its weird however that unrelated configuration options break boot... I have seen the issue already in the past (see #1203).

agners commented 3 years ago

It seems that this patch fixes the problem in my case: https://www.mail-archive.com/u-boot@lists.denx.de/msg409182.html

I'll add the patch to HAOS, so this should be resolved with the upcoming 6.3 release.