search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
5.04k stars 987 forks source link

Raspberry Pi3B+ + USB SSD Drive - Update to HassOS 5.8 fail #1095

Closed noide35 closed 1 year ago

noide35 commented 3 years ago

Hello,

just to Notify that I tried this night to upgrade my HA (Hass OS 4.17) installed on my PI3B+ to last Hass OS 5.8. I'm using my PI3B+ with a SSD drive connected on USB. This installation totally broke my setup : my raspberry became totally unresponsive 20 minutes waiting after the beginning of the upgrade. After that, I tried a fresh install of Hass OS 5.8 but also without success. Unfortunately I don't have any logs as I lost access to my PI and I have no screen connected to it.

I was able to get back a HA setup only by doing a fresh install of Hass OS 4.16 (the only one I had) and installing a snapshot on it.

So for now and from what I see, it seems there is some issues with this new Hass OS version and PI with USB boot setup.

Again sorry for not having any logs...

liudger commented 3 years ago

In my odroid n2 I have a similar problem. It is also stuck most of the times. Sometimes it finish booting but than minutes later the memory consumption spikes and the system is unresponsive again. Could be similar problem?

I'll try to get some logs on the next successful boot

agners commented 3 years ago

@noide35 you can download older releases from the GitHub releases page: https://github.com/home-assistant/operating-system/releases

noide35 commented 3 years ago

@agners Thanks, I missed that. It was late and I needed to quickly put a version running, so that was fine for me to go back to 4.16.

So for now I have Hass OS 4.16 with HA Core 2020.12.0 running well.

I also saw others issues with version 5.x of HA with PI and USB stick/drive connected. I think I'll have time during next weekend to do more tests and may be gather logs.

What I think I'll do :

sir106 commented 3 years ago

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz) - powering my pi 3b+ not even the green light flashed. after using a new flashed version of hassos_rpi3-64-4.16.img did the job. now my pi starts again... will try and update from a fresh installation... let's see if it works...

boesing commented 3 years ago

My Pi (4) does not even try to startup after upgrading 5.5 to 5.8. As I have a full USB-SSD installation, how can I install 5.5 or 5.7 without having to flash the whole SSD again? I probably have a backup but I would prefer having a rescue image without wiping the whole drive...

I wont update ever again after this 😅

Is there a way to use dd to clone a specific partition from a SD card to the SSD? đŸ€”

Edit: well, it seems that the update killed the whole partition table or boot partition as the pi does not even try to boot from USB SSD. Sadly I dont have an HDMI adapter and thus I am not able to see what the problem is. Maybe there is a way to just fix the boot partition with dd instead of flashing the whole system?

agners commented 3 years ago

@boesing I don't think its related the partition label. More likely you are hitting issue #1045: We deploy a new firmware for Raspberry Pi 4, and it seems to break USB SSD boot for some USB SSD enclosures.

The best way to recover is to plug-in the USB SSD to your computer, and browse the first partition (FAT partition). It should contain files like config.txt delete pieeprom.sig and pieeprom.upd from it. Then use Raspberry Pi OS to downgrade to an older Raspberry Pi firmware (e.g. 2020-09-03).

This issue is about Raspberry Pi 3.

fabiosci commented 3 years ago

Same problem. I use RPi 3 with USB SSD and it stopped working after lastest OS update. Connected to the TV I saw a black screen and "no signal" message on TV

agners commented 3 years ago

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz)

@sir106 you flashed the image for Raspberry Pi 4 and expected it to work on Raspberry Pi 3+?

agners commented 3 years ago

@fabiosci which version have you been using? 32-bit or 64-bit?

agners commented 3 years ago

Upgrade from 4.17 to 5.8 works and booted on my end using a Samsung 64GB USB flash drive on a Raspberry Pi 3+, so it seems this depends on the environment :-(

There could be several things which might be the culprit: Firmware, U-Boot or the new Linux Kernel 5.4. What would be interesting is to test different combinations of firmware/U-Boot with the 5.8 release: E.g. to test the old firmware, copy start.elf, fixup.dat and bootcode.bin from the first boot partition to the same place of a 5.8 USB drive. To test U-Boot from 4.17 copy u-boot.bin and boot.scr to the same place of a 5.8 USB drive.

When the screen turns black, it must be either the Raspberry Pi firmware or U-Boot which is causing issues. What would be interesting is to use the firmware

noide35 commented 3 years ago

Hello Stefan,

Thanks for this first analysis. I'll test ASAP what you're suggesting and will get back to you with informations.

To give you a little bit more info on my side, I'm using 32 bit image and when the issue appears, "ACT" LED of the Pi stays off (not blincking at all).

On my side I'm using these two parts : https://www.amazon.fr/gp/product/B011O9JAXS https://www.amazon.fr/gp/product/B01KFLH1WS

It's dirrectly connected to the Pi without external USB hub.

fabiosci commented 3 years ago

@fabiosci which version have you been using? 32-bit or 64-bit?

32bit

stanislav-brabec commented 3 years ago

I have problems with 5.8 as well. Upgrade from the web inteface ended with a heavily corrupted partition 8. Install to a fresh 32GB microSDHC card ended by a corrupted partition 8 as well, and additionally Card never left busy state, error -110. Returning back the original 16GB card, again corrupted partition 8. Instaliing fresh hassos_rpi3-4.20.img.gz works.

sir106 commented 3 years ago

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz)

@sir106 you flashed the image for Raspberry Pi 4 and expected it to work on Raspberry Pi 3+?

thanks for pointing it out!my fault... will try it again - was too quick when selecting the download ;)

wlatic commented 3 years ago

Have you tried the hdd in usb2 slot?

I've been having a very similar action and although the drives are recognized within raspberry pi (checked a few different USB caddys due to the issues some have), it'll boot when the SSD is in the USB2 slot and not in the USB3 slot.

May be a driver issue.

Have also seen issues with the power supply on the new Kernel. Before updating POE splitter was working fine, now it wants me to use my standard Pi Power Supply, otherwise there are issues.

noide35 commented 3 years ago

@wlatic I don't get your point there, Raspberry 3B+ has only USB2 ports and there no PoE at all in the loop as I'm using standard power supply (3A)

stanislav-brabec commented 3 years ago

I suspect that the source of all problems is a massive data corruption somewhere in the kernel. While trying to recover the crashed system, I have found that the first kilobyte of part8:supervisor/homeassistant/.storage/lovelace file is corrupted by a ~20 random single byte changes. Also the end of the part8:supervisor/homeassistant/home-assistant_v2.db is corrupted in a similar way.

The most often corruption bytes 0x00, 0x60, 0x7B, 0x22, 0x24, and less frequently other bytes.

noide35 commented 3 years ago

Hello everybody,

Hope you're doing well.

@agners You were right ! I've done som tests this morning and here are the results : 4.20 firmware (I never tested) => OK, working as expected 5.8 native (to test again and to be sure problem is still there) => KO ; one blink of ACT LED at power up, black screen and no more activity of ACT LED. 5.8 with firmware of 4.20 (start.elf, fixup.dat and bootcode.bin) => boot ok !

So it's seems to be an issue with firmware included in HasssOS 5.8

noide35 commented 3 years ago

Hello again,

I've just done one more test as I saw version 5.9 was available :

5.9 native (to be sure problem is still there) => KO ; one blink of ACT LED at power up, black screen and no more activity of ACT LED. 5.9 with firmware of 4.20 (start.elf, fixup.dat and bootcode.bin) => boot ok !

So same issue with HassOS 5.9.

GuruLarsson commented 3 years ago

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4
) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

pxFIN commented 3 years ago

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4
) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :)

After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

floow77 commented 3 years ago

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4
) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :)

After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

How did you do this ? "Just copy the boot files from the 5.3 FAT partition over 5.9" Because with hassio, we can't see the content of my ssd on my Mac.

eketjall commented 3 years ago

Can confirm I have the same issue not able to upgrade from the HA UI. In my case from 4.13 to 5.9 (no luck with a fresh 5.9 either) I'm running a RPi3B+ with a WD Green 120GB SSD and a StarTech USB312SAT3CB sata2usb adapter

rgreuel commented 3 years ago

Same issue here with a fresh 5.9 image running on a RPi3B with Kingston A400 and a StarTech USB312SAT3CB sata2usb adapter. Fails to boot.

Replaced start.elf, fixup.dat and bootcode.bin from the 4.20 image and now all is well with 5.9. Thanks for testing that @noide35

pxFIN commented 3 years ago

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4
) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :) After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

How did you do this ? "Just copy the boot files from the 5.3 FAT partition over 5.9" Because with hassio, we can't see the content of my ssd on my Mac.

I'm using HassOS installation.. so when you plug your USB SSD in Windows.. the boot partition 'mounts' and you can see all the RPI boot files.. basically you can also mount HassOS partitions different way.. run for example linux and use mount from command line

upwindanderl commented 3 years ago

Same for me here with a SanDisk Plus 120GB and a Renkforce 2,5 Case

Even with OS 5.10 no chance to boot without hacking, so i changed back to 4.20 stable https://github.com/home-assistant/operating-system/releases/tag/4.20

gregod commented 3 years ago

Here is a simple, reliable workaround using an shim bootloader on a SD card (referenced as Special bootcode.bin-only boot mode:

Format a SD card as FAT 32 and put the latest raspberry bootcode.bin on it.

The Pi will then boot from the sd card which will immediately hand over to the bootloader on the ssd. This is a brief read-only access so the normal reliability issues shouldn't apply. Since the sd card is never mounted and thus invisible to homeassistant this will survive upgrades.

Update: Other users have reported that this method does not work for them. It works for me, but is apparently not as reliable as i thought.

piffio commented 3 years ago

Hello everyone,

I have been suffering from the same issue reported here, but it took me a while to realise it was an issue with Home Assistant because I was just trying to move from SD cards to SSD over USB on a new USB adaptor. At first I thought it was a problem with the adaptor itself, but then I found out that Raspbbian (Raspios) was booting properly from the same disk / connector.

I am running on RPI 3B (not plus) and I did the required OTP changes to enable USB boot. The interesting thing was that either 5.9 or 5.10 didn't manage to boot most of the time, but sometimes and quite randomly the Pi was booting up successfully. I can confirmed that since switching to 5.3 the USB boot works reliably every time, without the need to have a special bootloader on an SD card.

During the troubleshooting I hooked up a UART serial cable to the RPI and was booting with a modified bootloader.bin from an SD card in order to enable the UART console during the boot sequence, and foun the following.

This is the log from a successful boot:

Raspberry Pi Bootcode

Read File: config.txt, 1827

Read File: start.elf, 2949600 (bytes)

Read File: fixup.dat, 7289 (bytes)

MESS:00:00:04.891066:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:04.895956:0: brfs: File read: 1827 bytes
MESS:00:00:04.948950:0: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:04.955112:0: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:04.961449:0: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:04.967786:0: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:04.974123:0: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:04.980459:0: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:04.986797:0: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:04.993133:0: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:04.999470:0: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.005807:0: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.011902:0: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.043204:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.048023:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.072746:0: brfs: File read: 1827 bytes
MESS:00:00:05.266608:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:05.274060:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.280084:0: *** Restart logging
MESS:00:00:05.284683:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.292052:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.298910:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.305769:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.312625:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.319483:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.326342:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.333198:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.340056:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.346915:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.353529:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.359432:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.367226:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.374084:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.380942:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.387800:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.394657:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.401515:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.408372:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.415230:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.422087:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.428703:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.434576:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:05.443064:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:05.448756:0: vec: vec_middleware_power_on: vec_base: 0x7e806000 rev-id 0x00002708 @ vec: 0x7e806100 @ 0x00000420 enc: 0x7e806060 @ 0x00000220 cgmsae: 0x7e80605c @ 0x00000000
MESS:00:00:05.470073:0: dtb_file 'bcm2710-rpi-3-b.dtb'
MESS:00:00:05.513395:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:05.518058:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x100 size 0x6d40
MESS:00:00:05.538014:0: brfs: File read: 27968 bytes
MESS:00:00:06.088722:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.093246:0: dtparam: audio=on
MESS:00:00:06.112323:0: brfs: File read: 1827 bytes
MESS:00:00:06.128466:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:06.132430:0: Read command line from file 'cmdline.txt':
MESS:00:00:06.138317:0: 'dwc_otg.lpm_enable=0 console=tty1'
MESS:00:00:06.159418:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:07.628019:0: brfs: File read: 34 bytes
MESS:00:00:07.773572:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:07.777435:0: Loading 'u-boot.bin' to 0x8000 size 0x61f0c
MESS:00:00:07.783434:0: Device tree loaded to 0x2eff8e00 (size 0x7118)
MESS:00:00:07.790695:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:07.797357:0: uart: Baud rate change done...
MESS:00:00:07.800771:0: uart: Baud rate

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
switch to partitions #0, OK
mmc0 is current device
** No partition table - mmc 0 **
Card did not respond to voltage select!

Device 0: Vendor: ASMT     Rev: 0    Prod: 2115            
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
Found U-Boot script /boot.scr
2421 bytes read in 1 ms (2.3 MiB/s)
## Executing script at 02400000

Device 0: Vendor: ASMT     Rev: 0    Prod: 2115            
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
loading env...

usb read: device 0 block # 1214464, count 32 ... 32 blocks read: OK
## Error: bad CRC, import failed
Trying to boot slot A, 2 attempts remaining. Loading kernel ...
7639672 bytes read in 191 ms (38.1 MiB/s)
storing env...

usb write: device 0 block # 1214464, count 32 ... 32 blocks written: OK
Starting kernel
Kernel image @ 0x080000 [ 0x000000 - 0x749278 ]
## Flattened Device Tree blob at 2eff8e00
   Booting using the fdt blob at 0x2eff8e00
   Using Device Tree in place at 2eff8e00, end 2f002f17

Starting kernel ...

This is the log from an unsuccessful boot:

Raspberry Pi Bootcode

Read File: config.txt, 1827

Read File: start.elf, 2949600 (bytes)

Read File: fixup.dat, 7289 (bytes)

MESS:00:00:05.811133:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.816025:0: brfs: File read: 1827 bytes
MESS:00:00:05.921275:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.926099:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.950823:0: brfs: File read: 1827 bytes
MESS:00:00:06.146568:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:06.154020:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:06.160044:0: *** Restart logging
MESS:00:00:06.196166:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:06.205444:0: dtb_file 'bcm2710-rpi-3-b.dtb'
MESS:00:00:06.248811:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:06.253471:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x100 size 0x6d40
MESS:00:00:06.273171:0: brfs: File read: 27968 bytes
MESS:00:00:06.825014:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.829550:0: dtparam: audio=on
MESS:00:00:06.848617:0: brfs: File read: 1827 bytes
MESS:00:00:06.864757:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:06.868726:0: Read command line from file 'cmdline.txt':
MESS:00:00:06.874615:0: 'dwc_otg.lpm_enable=0 console=tty1'
MESS:00:00:06.895747:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:08.372110:0: brfs: File read: 34 bytes
MESS:00:00:08.517615:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:08.521476:0: Loading 'u-boot.bin' to 0x8000 size 0x61f0c
MESS:00:00:08.527475:0: Device tree loaded to 0x2eff8e00 (size 0x711c)
MESS:00:00:08.534745:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:08.541402:0: uart: Baud rate change done...
MESS:00:00:08.544813:0: uart: Baud rate

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
switch to partitions #0, OK
mmc0 is current device
** No partition table - mmc 0 **
Card did not respond to voltage select!

Device 0: unknown device
Waiting for Ethernet connection... unable to connect.
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-b8-27-eb-67-f6-d4
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcm283x-rpi
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcm283x
Waiting for Ethernet connection... 

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
WARNING at drivers/mmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
WARNING at drivers/mmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
Card did not respond to voltage select!
Card did not respond to voltage select!

Device 0: unknown device
Waiting for Ethernet connection... unable to connect.
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-b8-27-eb-67-f6-d4

Basically it looks like U-Boot only saw 3 USB devices on the failed attempt, while the successful attempt reported 4 USB devices, including the SSD drive. After the PXE boot attempt time out I got an HassOS> prompt where I tried to issue an USB reset multiple times, all of them failing to detect the attached drive:

HassOS> usb reset
resetting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

The following is the output from usb tree

HassOS> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |   U-Boot Root Hub 
  |
  |+-2  Hub (480 Mb/s, 2mA)
    |
    |+-3  Vendor specific (480 Mb/s, 2mA)

But then I tried a usb stop followed by a usb start and here is what happened:

HassOS> usb stop
stopping USB..
HassOS> use   b start
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
HassOS> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |   U-Boot Root Hub 
  |
  |+-2  Hub (480 Mb/s, 2mA)
    |
    |+-3  Vendor specific (480 Mb/s, 2mA)
    |  
    |+-4  Mass Storage (480 Mb/s, 0mA)
         ASMedia AS2115 00000000000000000000

Following is the output from usb info after reinitializing the USB stack

1: Hub,  USB Revision 1.10
 -  U-Boot Root Hub 
 - Class: Hub
 - PacketSize: 8  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 0.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 2 Interval 255ms

2: Hub,  USB Revision 2.0
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0424  Product 0x9514 Version 2.0
   Configuration: 1
   - Interfaces: 1 Self Powered Remote Wakeup 2mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms

3: Vendor specific,  USB Revision 2.0
 - Class: Vendor specific
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0424  Product 0xec00 Version 2.0
   Configuration: 1
   - Interfaces: 1 Self Powered Remote Wakeup 2mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 3
     - Class Vendor specific
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512
     - Endpoint 3 In Interrupt MaxPacket 16 Interval 4ms

4: Mass Storage,  USB Revision 2.10
 - ASMedia AS2115 00000000000000000000
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x174c  Product 0x1153 Version 0.1
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

I then tried to run usbboot without success because I could not figure out a way to select the device to boot from, and could not really find any online documentation.

Given all of the above I would tend to think that the issue is with U-Boot, and I hope that the debugging data might be useful to track down the issue. Now the device is happily running on 5.3 and attached to my production instance of Home Assistant, but let me know if I can provide you more data to help troubleshooting

eketjall commented 3 years ago

Here is a simple, reliable workaround using an shim bootloader on a SD card (referenced as Special bootcode.bin-only boot mode:

Format a SD card as FAT 32 and put the latest raspberry bootcode.bin on it.

The Pi will then boot from the sd card which will immediately hand over to the bootloader on the ssd. This is a brief read-only access so the normal reliability issues shouldn't apply. Since the sd card is never mounted and thus invisible to homeassistant this will survive upgrades.

Just tried this with a fresh 5.10 image on the ssd but still no boot i'm affraid.

eketjall commented 3 years ago

Upgrade from 4.17 to 5.8 works and booted on my end using a Samsung 64GB USB flash drive on a Raspberry Pi 3+, so it seems this depends on the environment :-(

There could be several things which might be the culprit: Firmware, U-Boot or the new Linux Kernel 5.4. What would be interesting is to test different combinations of firmware/U-Boot with the 5.8 release: E.g. to test the old firmware, copy start.elf, fixup.dat and bootcode.bin from the first boot partition to the same place of a 5.8 USB drive. To test U-Boot from 4.17 copy u-boot.bin and boot.scr to the same place of a 5.8 USB drive.

When the screen turns black, it must be either the Raspberry Pi firmware or U-Boot which is causing issues. What would be interesting is to use the firmware

I've tried this in different combinations using boot files from 4.20 on an 5.10 image 1 - start.elf, fixup.dat and bootcode.bin 2 - u-boot.bin and boot.scr 3 - 1 and 2 at the same time.

Still no boot from ssd.

If all the boot files, incl overlays folder is used from 4.20 then it works.

JustAmply commented 3 years ago

Here is a simple, reliable workaround using an shim bootloader on a SD card (referenced as Special bootcode.bin-only boot mode: Format a SD card as FAT 32 and put the latest raspberry bootcode.bin on it. The Pi will then boot from the sd card which will immediately hand over to the bootloader on the ssd. This is a brief read-only access so the normal reliability issues shouldn't apply. Since the sd card is never mounted and thus invisible to homeassistant this will survive upgrades.

Just tried this with a fresh 5.10 image on the ssd but still no boot i'm affraid.

Yeah, i couldn't get it to work either. Thought i misunderstood something...

Crashing a few times every day on HassOS 5.3 and core-2021.1.3.

floow77 commented 3 years ago

Same issue for me. I've tried with bootcode.bin on a SD card but the upgrade to v5.10 it's always KO.

timmmmmmmmm commented 3 years ago

I've tried this in different combinations using boot files from 4.20 on an 5.10 image 1 - start.elf, fixup.dat and bootcode.bin 2 - u-boot.bin and boot.scr 3 - 1 and 2 at the same time.

Still no boot from ssd.

If all the boot files, incl overlays folder is used from 4.20 then it works.

Do you mean that eventually you got it to boot from SSD? By copying all (?) boot files from the 4.2 image on the 5.1 boot? I also just tried some combinations of files, but unfortunately it's still not booting.

eketjall commented 3 years ago

I've tried this in different combinations using boot files from 4.20 on an 5.10 image 1 - start.elf, fixup.dat and bootcode.bin 2 - u-boot.bin and boot.scr 3 - 1 and 2 at the same time. Still no boot from ssd. If all the boot files, incl overlays folder is used from 4.20 then it works.

Do you mean that eventually you got it to boot from SSD? By copying all (?) boot files from the 4.2 image on the 5.1 boot? I also just tried some combinations of files, but unfortunately it's still not booting.

Yes. Works for me after coping ALL the boot files/folders from 4.20 onto the 5.10 boot partition.

maovidal commented 3 years ago

I've tried this in different combinations using boot files from 4.20 on an 5.10 image 1 - start.elf, fixup.dat and bootcode.bin 2 - u-boot.bin and boot.scr 3 - 1 and 2 at the same time. Still no boot from ssd. If all the boot files, incl overlays folder is used from 4.20 then it works.

Do you mean that eventually you got it to boot from SSD? By copying all (?) boot files from the 4.2 image on the 5.1 boot? I also just tried some combinations of files, but unfortunately it's still not booting.

Yes. Works for me after coping ALL the boot files/folders from 4.20 onto the 5.10 boot partition.

Thank you! It worked for me copying all files from a 4.20 to an SD card which is only for booting (instead rewriting the content of the 5.10 SSD boot partition).

I have a question regarding upgrades. Does this combo of 4.20 boot files and 5.X could be an issue?

JustAmply commented 3 years ago

I just tried updating to 5.11 and am happy to report that it boots properly for me (after a quick power cycle). Had no problems with the 4.20 files copied onto the 5.10 boot partition until now. But 5.11 boots a lot quicker!

Just have to see if it's a stable solution. Fingers crossed!

upwindanderl commented 3 years ago

I just tried updating to 5.11 and am happy to report that it boots properly for me (after a quick power cycle). Had no problems with the 4.20 files copied onto the 5.10 boot partition until now. But 5.11 boots a lot quicker!

Just have to see if it's a stable solution. Fingers crossed!

you did some additional copy action as well or only the 5.11 os and done?

floow77 commented 3 years ago

Thank you ! It's works !! I have copied the 4.20 files without overlays folder on my SD card and the version 5.11 boots with success on my SSD

JustAmply commented 3 years ago

I just tried updating to 5.11 and am happy to report that it boots properly for me (after a quick power cycle). Had no problems with the 4.20 files copied onto the 5.10 boot partition until now. But 5.11 boots a lot quicker! Just have to see if it's a stable solution. Fingers crossed!

you did some additional copy action as well or only the 5.11 os and done?

My old working setup was 5.10 with the 4.20 boot files copied over. And now i've just updated to 5.11 via the OTA update button in the supervisor. Did nothing else.

It might and should (maybe) work if you set it up completely from scratch with 5.11, but i haven't tested this.

rgreuel commented 3 years ago

I just tried updating to 5.11 and am happy to report that it boots properly for me (after a quick power cycle). Had no problems with the 4.20 files copied onto the 5.10 boot partition until now. But 5.11 boots a lot quicker! Just have to see if it's a stable solution. Fingers crossed!

you did some additional copy action as well or only the 5.11 os and done?

My old working setup was 5.10 with the 4.20 boot files copied over. And now i've just updated to 5.11 via the OTA update button in the supervisor. Did nothing else.

It might and should (maybe) work if you set it up completely from scratch with 5.11, but i haven't tested this.

Are you using an SD card to hold the boot files too?

I am still on 5.9 with the 4.20 boot files (no sd card), and couldn't boot after upgrading to 5.10 using supervisor. Haven't yet tried 5.11 but wondering if I'll need an sd card with the old boot files to be able to boot.

JustAmply commented 3 years ago

Are you using an SD card to hold the boot files too?

I am still on 5.9 with the 4.20 boot files (no sd card), and couldn't boot after upgrading to 5.10 using supervisor. Haven't yet tried 5.11 but wondering if I'll need an sd card with the old boot files to be able to boot.

Oh yeah, my bad. I failed to mention that i am booting using just my SSD with a UGREEN USB 3.0 to SATA Adapter from Amazon. I have no MicroSD installed, just the SSD with 5.11!

bcutter commented 3 years ago

Are you using an SD card to hold the boot files too? I am still on 5.9 with the 4.20 boot files (no sd card), and couldn't boot after upgrading to 5.10 using supervisor. Haven't yet tried 5.11 but wondering if I'll need an sd card with the old boot files to be able to boot.

Oh yeah, my bad. I failed to mention that i am booting using just my SSD with a UGREEN USB 3.0 to SATA Adapter from Amazon. I have no MicroSD installed, just the SSD with 5.11!

As I just stumbled about this issue during some research on how to switch my HASS OS on a Pi 4 from SD card to a SSD (having same UGREEN USB 3.0 adapter, case version): is it necessary to perform some "copy old boot files from an earlier version like 4.20 to the SSD" steps or is this because of some troubles specifically with the Pi 3B+?

pxFIN commented 3 years ago

Are you using an SD card to hold the boot files too? I am still on 5.9 with the 4.20 boot files (no sd card), and couldn't boot after upgrading to 5.10 using supervisor. Haven't yet tried 5.11 but wondering if I'll need an sd card with the old boot files to be able to boot.

Oh yeah, my bad. I failed to mention that i am booting using just my SSD with a UGREEN USB 3.0 to SATA Adapter from Amazon. I have no MicroSD installed, just the SSD with 5.11!

As I just stumbled about this issue during some research on how to switch my HASS OS on a Pi 4 from SD card to a SSD (having same UGREEN USB 3.0 adapter, case version): is it necessary to perform some "copy old boot files from an earlier version like 4.20 to the SSD" steps or is this because of some troubles specifically with the Pi 3B+?

This is specifically for the RPI 3B/3B+.

RPI4 has firmware and there has been problems with it (=SSD booting).. Take a look at other issue threads.. there are plenty about RPI4

Also you could take a look here https://github.com/raspberrypi/firmware/issues

I tried RPI4 SSD booting in the end of last year and noticed booting problems/weirdness but havent tried after that so dont know if the situation is better so sorry cant help you in this matter

RPI3B -> I'm still running 5.9 OS (with 5.3 boot files copied over).. My finger is itching to try 5.11 but.. maybe I'll wait :)

stanislav-brabec commented 3 years ago

I have RPi 3B+, boot from SD (Eaget 32 GB class A1). 4.20: Worked perfectly. 5.8: Failed during installation. Second attempt to install succeeded. Failed during boot. Total data loss. 5.9: Failed during boot. Total data loss. 5.10: Failed after few hours. It was rebootable 2 times, 3rd time it was fatal, total data loss. 5.11: No problem in last 36 hours. I am running unmodified 5.11.

So I am starting to hope that the upstream kernel update fixed it.

fabiosci commented 3 years ago

I have RPi 3B+, boot from SD (Eaget 32 GB class A1). 4.20: Worked perfectly. 5.8: Failed during installation. Second attempt to install succeeded. Failed during boot. Total data loss. 5.9: Failed during boot. Total data loss. 5.10: Failed after few hours. It was rebootable 2 times, 3rd time it was fatal, total data loss. 5.11: No problem in last 36 hours. I am running unmodified 5.11.

So I am starting to hope that the upstream kernel update fixed it.

I'm currently running 4.20 and decided to avoid to upgrade after a total data loss like yours, recovered thanks to a snapshot. Did you reinstalled 4.20 and then upgraded directly to 5.11? I'd like to understand if running the upgrade process now from 4.20 directly to 5.11 without any other manual activity is safe or not.

thanks

JustAmply commented 3 years ago

I'm currently running 4.20 and decided to avoid to upgrade after a total data loss like yours, recovered thanks to a snapshot. Did you reinstalled 4.20 and then upgraded directly to 5.11? I'd like to understand if running the upgrade process now from 4.20 directly to 5.11 without any other manual activity is safe or not.

thanks

I would just make and download a snapshot and then upgrade. For me it worked without any tinkering. Still very stable!

stanislav-brabec commented 3 years ago

Did you reinstalled 4.20 and then upgraded directly to 5.11? I'd like to understand if running the upgrade process now from 4.20 directly to 5.11 without any other manual activity is safe or not.

Yes, I did. Just clicked to Upgrade in the web browser.

eketjall commented 3 years ago

I just tried updating to 5.11 and am happy to report that it boots properly for me (after a quick power cycle). Had no problems with the 4.20 files copied onto the 5.10 boot partition until now. But 5.11 boots a lot quicker! Just have to see if it's a stable solution. Fingers crossed!

you did some additional copy action as well or only the 5.11 os and done?

My old working setup was 5.10 with the 4.20 boot files copied over. And now i've just updated to 5.11 via the OTA update button in the supervisor. Did nothing else.

It might and should (maybe) work if you set it up completely from scratch with 5.11, but i haven't tested this.

I had 5.10 running with 4.20 boot-files Did an upgrade to 5.11 from the supervisor. Unfortunately it did not boot after that . Tried to just copy the 4.20 boot files to the non booting 5.11 Still no boot. Then did a fresh 5.11 from image but still no boot. Copied the 4.20 boot files top the fresh 5.11 an finally got it booted. So, for me it is the same problem also with 5.11 as befor.

Don't forget to take a snapshot before every update attempt.

stanislav-brabec commented 3 years ago

I have RPi 3B+, boot from SD (Eaget 32 GB class A1). 5.11: No problem in last 36 hours. I am running unmodified 5.11. So I am starting to hope that the upstream kernel update fixed it.

Update: Two system freezes experienced in the last two weks. I had to forcefully power off my RPi to reboot. Symptoms were:

So, 5.11 is more stable than 5.10, but not as stable as 4.20.

I don't know, how to get pre-freeze logs.

joostlitjens commented 3 years ago

Thanks too this topic i found out why 5.12 does not boot

I have copied the 4.20 boot files without overlays folder on my SSD and the version 5.12 boots with success on my SSD (without SD) I'm using Pi3B+