raspberrypi / rpi-eeprom

Installation scripts and binaries for the Raspberry Pi 4 and Raspberry Pi 5 bootloader EEPROMs
https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#raspberry-pi-boot-eeprom
Other
1.27k stars 203 forks source link

SOLVED: SSD Enclosures with Jmicron chips JMS580 and JMS583 don't boot up. RPi4. #266

Closed carras closed 3 years ago

carras commented 3 years ago

Hi there:

With the latest EEPROM firmware. I've try critical, stable and beta channels. My SSD enclosure from ORICO using the chip JMS580 from Jmicron doesn't boot up. And I've seen other people with the same issue with enclosure with JMS580.

If we plug the raspi on and after we connect the ssd, then it boots up.

And It was working before, but the latest critical and stable don't work now.

PD: I want to boot up HassOS form the SSD, but the implemented an auto firmware (EEPROM) update that it's always going to prevent me to use an older firmware.

Update: I add JMS578 too, to de list of controllers with the same issue.

TRY THIS SOLUTION:

Solution: with the command "sudo -E rpi-eeprom-config --edit" you edit the EEPROM BOOTLOADER then you the next configuration: USB_MSD_PWR_OFF_TIME=0

Holland1 commented 3 years ago

Can confirm there is a similar issue with the JMS583 controller.

caro7372 commented 3 years ago

I have exactly the same problem as above described.

timg236 commented 3 years ago

Please can you add screenshots showing the failure on the bootloader HDMI diagnostics screen or UART traces.

lurch commented 3 years ago

I want to boot up HassOS form the SSD, but the implemented an auto firmware (EEPROM) update that it's always going to prevent me to use an older firmware.

You could try setting the FREEZE_VERSION option as a workaround to stop that? https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

timg236 commented 3 years ago

Yes FREEZE_VERSION will prevent upgrades except for rescue image and because it's in the EEPROM will work if you swap SD cards.

The most likely explanation is that the controller is fussy about exactly how long USB port power is off for and any delays during initialisation. However, we'd need screenshots / logs to confirm.

N.B. Please can you also confirm that when booted from SD card the enclosure works reliably when mounted under Linux and will hotplug correctly if you connect/disconnect the cable without pressing any special power buttons on the enclosure?

carras commented 3 years ago

@timg236 This is it, but it's not Raspberry Pi OS. I can do it with it if you ask, this is HassOS. Is this ok?

IMG_0329

The disk enclosure Is visible when you boot Raspberry Pi OS and remember "If we plug the raspi on and after we connect the ssd, then it boots up."

carras commented 3 years ago

I add JMS578 too, to de list of controllers with the same issue. As being reported by other user.

carras commented 3 years ago

I just contacted Jmicron in case the need something to add, or they can help with the next message:

When try to boot up from USB enclosure on a Raspberry Pi 4 your controllers JMS578, JMS580 and JMS583 are reported to be not working properly.

I've open an issue in the Github page of the EEPROM firmware of the Raspberry Pi 4: https://github.com/raspberrypi/rpi-eeprom/issues/266

There are many Raspberry Pi 4 users in world that would like to boot up with USB on a SSD and your chips are in many cases like ORICO (which I have two) or Kingspecs. As te result of your controller no working those brands and your brand is losing popularity over Realtech or ASM with apparently work better.

Could you please investigate the issue and help the Raspberry Pi team to work on this problem?

Thanks.

timg236 commented 3 years ago

I doubt this is anything to do with the JMicron chip because there are plenty of people on the forums using JMicron USB SATA adapters. Some of the older chips have issues with UAS but that's not relevant until Linux runs. I think the issue is with the design of the enclosure.

caro7372 commented 3 years ago

Please can you add screenshots showing the failure on the bootloader HDMI diagnostics screen or UART traces.

A3A15FA4-135E-4693-A6C4-A3EBF24E7857

caro7372 commented 3 years ago

Yes FREEZE_VERSION will prevent upgrades except for rescue image and because it's in the EEPROM will work if you swap SD cards.

The most likely explanation is that the controller is fussy about exactly how long USB port power is off for and any delays during initialisation. However, we'd need screenshots / logs to confirm.

N.B. Please can you also confirm that when booted from SD card the enclosure works reliably when mounted under Linux and will hotplug correctly if you connect/disconnect the cable without pressing any special power buttons on the enclosure?

Yes, SD card works! And cable SSD out and in again boot works.

timg236 commented 3 years ago

Please post links to the SSD enclosure. If it's still for sale and we should be able to get one for some interop testing in the new year. Please also confirm the exact EEPROM version that worked for you. It may be that a configurable timeout quirk could be added to support such devices.

Holland1 commented 3 years ago

The one with the JMS583 chip

https://www.amazon.nl/dp/B07QN2WFGB/ref=pe_19967891_404437601_TE_item

caro7372 commented 3 years ago

https://www.aliexpress.com/item/4001278381663.html

vodoc commented 3 years ago

Small investigation of my SSD (based on JMS583) https://aliexpress.ru/item/4000402460407.html EEPROM versions: Thu 3 Sep 12:11:43 UTC 2020 (1599135103) - boots ok. Wed 28 Oct 17:32:40 UTC 2020 (1603906360) - boots only after unplug + plug again Fri 11 Dec 11:15:17 UTC 2020 (1607685317) - boots only after unplug + plug again

carras commented 3 years ago

I've got this bundle which I think its a really good quality/price: https://www.aliexpress.com/item/1005001413198976.html?spm=a2g0s.9042311.0.0.27424c4dTdwJ3z

It uses JMS580 and a SATA m.2 SSD.

carras commented 3 years ago

@caro7372 I have this Orico enclosure with the JMS578 but it has the format of 2.5 inches disks. And it works perfectly to boot up with the Raspberry. Can you check with other ssd blade? Nevermind, you can not open your enclosure. https://www.aliexpress.com/item/4001146259311.html?spm=a2g0s.9042311.0.0.27424c4dc8LdKF *In that enclosure I put a Samsung 540 SSD.

timg236 commented 3 years ago

Has anyone tried increasing the USB port power-off time by setting USB_MSD_PWR_OFF_TIME=5000 in the EEPROM config? This should look identical to a disconnect to the device. The default off-time may be less between 3rd September and subsequent releases.

caro7372 commented 3 years ago

I will check and try! I had to find out how to edit config.txt but I managed it. So I set USB_MSD_PWR_OFF_TIME=5000 there in the config.txt Reboot without SD-card so SSD should boot but no succes.

carras commented 3 years ago

Has anyone tried increasing the USB port power-off time by setting USB_MSD_PWR_OFF_TIME=5000 in the EEPROM config? This should look identical to a disconnect to the device. The default off-time may be less between 3rd September and subsequent releases.

Well, this made the trick. I didn't make it 5000 I made it 0. Then it works. I've tried your 5000 suggestion but It didn't work.

Solution: with the command "sudo -E rpi-eeprom-config --edit" you edit the EEPROM BOOTLOADER then you the next configuration: USB_MSD_PWR_OFF_TIME=0

@timg236 I guess there is a reason for it. How much USB_MSD_PWR_OFF_TIME=1000 is necesary? Is it bad to have it at 0? Why is important to cut the power to the USB for some time before it boots?

From release notes: "USB ports are now powered off for 1 second at startup regardless of boot mode to resolve issues with some USB devices which failed after a reboot. This can be configured via the existing USB_MSD_PWR_OFF_TIME EEPROM setting." https://github.com/raspberrypi/rpi-eeprom/releases/tag/v2020.07.31-138a1

The documentation can be found: https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

caro7372 commented 3 years ago

@carras Where did you set that 0 ms? In the config.txt ( on the SD-card ) or ???? I'm a bit confused now if I did the right thing.

carras commented 3 years ago

@carras Where did you set that 0 ms? In the config.txt ( on the SD-card ) or ???? I'm a bit confused now if I did the right thing.

You get an SD-card with Raspberry Pi OS and boot with it. Then you execute the command in SSH/terminal: sudo -E rpi-eeprom-config --edit You are now in one text file and you need to add this at the end of the list: USB_MSD_PWR_OFF_TIME=0

caro7372 commented 3 years ago

@carras Where did you set that 0 ms? In the config.txt ( on the SD-card ) or ???? I'm a bit confused now if I did the right thing.

You get an SD-card with Raspberry Pi OS and boot with it. Then you execute the command in SSH/terminal: sudo -E rpi-eeprom-config --edit You are now in one text file and you need to add this at the end of the list: USB_MSD_PWR_OFF_TIME=0

What I see know is that the 2020-10-28 has no stable release? 2020-12-11 is latest stable. You see that too?

After applying the USB ...... to 0 you reboot with or without the SD card in ( for the first time ? ). Sorry for my noob questions but I'm relatively new to Linux.

--> Okay, you nailed it. I was setting it on the wrong place. Once you pointed me in the right direction it is working!!

timg236 commented 3 years ago

USB port power is not guaranteed across a reboot, it goes away briefly on 2G, 4G during reset and for much longer on 8G. This is the same as a disconnect and for devices such as spinning hard drive enclosures, it's better to have a long power off than a short one. For an NVMe SSD it should be fine to have USB_MSD_PWR_OFF_TIME=0 because it will power on almost immediately.

So changes to improve compatibility for spinning hard drive enclosures seems to be worse for some JMicron NVME adapters. There might be a common change which is good for both but NVME enclosures have been fairly low down the compatibility list because there's no performance benefit of USB SATA SSD

timg236 commented 3 years ago

@carras Where did you set that 0 ms? In the config.txt ( on the SD-card ) or ???? I'm a bit confused now if I did the right thing.

You get an SD-card with Raspberry Pi OS and boot with it. Then you execute the command in SSH/terminal: sudo -E rpi-eeprom-config --edit You are now in one text file and you need to add this at the end of the list: USB_MSD_PWR_OFF_TIME=0

What I see know is that the 2020-10-28 has no stable release? 2020-12-11 is latest stable. You see that too?

After applying the USB ...... to 0 you reboot with or without the SD card in ( for the first time ? ). Sorry for my noob questions but I'm relatively new to Linux.

2020-10-28 is the version number of a beta release. Stable releases are a subset of beta releases and default releases (and critical updates) are a subset of stable releases. Therefore, if you configure your system to track the beta releases you'll see all the images, some good, some bad.

caro7372 commented 3 years ago

@carras Where did you set that 0 ms? In the config.txt ( on the SD-card ) or ???? I'm a bit confused now if I did the right thing.

You get an SD-card with Raspberry Pi OS and boot with it. Then you execute the command in SSH/terminal: sudo -E rpi-eeprom-config --edit You are now in one text file and you need to add this at the end of the list: USB_MSD_PWR_OFF_TIME=0

What I see know is that the 2020-10-28 has no stable release? 2020-12-11 is latest stable. You see that too? After applying the USB ...... to 0 you reboot with or without the SD card in ( for the first time ? ). Sorry for my noob questions but I'm relatively new to Linux.

2020-10-28 is the version number of a beta release. Stable releases are a subset of beta releases and default releases (and critical updates) are a subset of stable releases. Therefore, if you configure your system to track the beta releases you'll see all the images, some good, some bad.

Yes, I understand that ( now ) but was a bit confused about Home Assistant 5.8 was setting it to 2020-10-28 although it now seems it was Beta.

carras commented 3 years ago

@caro7372 There are 3 types of releases that you can subscribe to: "critical" "stable" and "beta". I recommend you use the command "sudo raspi-config" go to advance options, then bootloader version, and the select latest. Then you need to edit the file I told you about.

Or just do the file thing, because HassOS will update the firmware for you apparently.

timg236 commented 3 years ago

It's up to the Home Assistant maintainers to decide which bootloader updates the OS installs. Tracking beta by default seems rather risky because the whole point of beta is to find bugs!

carras commented 3 years ago

There might be a common change which is good for both but NVME enclosures have been fairly low down the compatibility list because there's no performance benefit of USB SATA SSD

@timg236 Yes, but I use a SATA m.2 enclosure, no NVMe. But as I said before I feel this issue is only on Jmicron controllers. So do I have to update the config file with every update? *I've tested that it doesn't work with "critical" updates, only from stable.

caro7372 commented 3 years ago

@caro7372 There are 3 types of releases that you can subscribe to: "critical" "stable" and "beta". I recommend you use the command "sudo raspi-config" go to advance options, then bootloader version, and the select latest. Then you need to edit the file I told you about.

Or just do the file thing, because HassOS will update the firmware for you apparently.

Yes, that part I'm ( was ) aware of. So I set it back to "stable". I'm setting up the system again!

vodoc commented 3 years ago

I may confirm that USB_MSD_PWR_OFF_TIME=0 works fine for my SSD (based on JMS583) on latest beta Fri 11 Dec 11:15:17 UTC 2020 (1607685317) It if not fixes boot issue for HassOS (it replaces config each boot), but looks like we may have solution and fix in the future.

timg236 commented 3 years ago

@vodoc Thanks for the tests. I've requested this and some other devices for the interop test but that won't happen until the New Year. Hopefully, we'll find something which works with both these drives and the library of awkward devices :)

N.B. If HassOS is using rpi-eeprom-update then it should respect FREEZE_VERSION=1 in the bootloader EEPROM config and leave the bootloader alone.

carras commented 3 years ago

@vodoc Wait a moment, HassOS it does work with me. On reboot, soft reboot and turn off and on. Check the issue there: https://github.com/home-assistant/operating-system/issues/1045

@timg236 Ok, let's assume FREEZE_VERSION=1 works. But do I lost the config file every time it updates? Can I prevent the config file to be modify?

And, another question, I put FREEZE_VERSION=1 but I deleted later, the documentation said something about how to disable different that remove it from config file. Do I have to do another thing to disable it and that's why @vodoc can't reboot without resetting the config file?

Documentation: FREEZE_VERSION

Previously this property was only checked by the rpi-eeprom-update script. However, now that self-update is enabled the bootloader will also check this property. If set to 1, this overrides ENABLE_SELF_UPDATE to stop automatic updates. To disable FREEZE_VERSION you will have to use an SD card boot with recovery.bin.

agners commented 3 years ago

@caro7372 @timg236 yeah we were debating if we should use beta, but we thought since it fixes known issues of external hard drives (and the update only applies if it is on a USB hard drive), it make sense to take the plunge (see discussion here https://github.com/home-assistant/operating-system/pull/939).

Maybe it was a bad decision... But then, the latest stable would now have broken JMS580/JMS583 too :man_shrugging:

In the end, I am inclined to just remove firmware update entirely, then users can figure out what firmware/USB SSD combinations work using Raspberry Pi OS, and then switch to Home Assistant OS once they have a working firmware/USB SSD combination.

caro7372 commented 3 years ago

Goodmorning, I read the discussion about auto update the eeprom. Only thing I am curious about is why HomeAssistant wants to update this. Iā€™m sure there will be benefit of doing it but is it really needed?

Regards,

Robert

p4p3r commented 3 years ago

In case it's useful for future testing, here's a few enclosures I tested (RPi4, HassOS 64bit).

Booting fine:

Booting only after unplug/plug:

THX1180 commented 3 years ago

Having a similar issue with my Orico bh100 external SSD (JMicron JMS567 SATA 6Gb/s) and opened a thread on the official forums...

Here's what a typed out on the forum:

Hi Raspberry Pi community,

I bought a powered hub ( TP-LINK UH720) as recommended in the storage quirks thread) to boot with my external SSD (Orico bh100), because i was heaving power issues connecting the SSD directly to the Pi. I reinstalled everything from scratch and everything booted up fine, and in fact worked like a charm for weeks on end.

No error messages or strange behavior whatsoever...

Updated the system a couple of days ago and it's no longer functioning properly (see the included images). I know the SSD is working fine because, i have extensively tested it on the PC (manjaro) and it was functioning properly for weeks. The Pi i directly connected and powered by the hub and it's getting enough juice, because i can easily connected other external hdd's etc... But SSD will often just not even register, the hub has little lights (indicators of connection) and it will just blink quickly and then do nothing. It will sporadically boot up into the system when i power it on/off randomly 6 to10 times but that's about it, when i try to reboot i get the error messages and it won't do jacks$#5

It will just hang displaying these messages (click to enlarge the images):

20201216-145348 20201216-145353

I already did a fresh install and add the quirks, and even tried manjaro instead of Pi Os. But nothing and i mean nothing seems to work, and if it works it will stop working when rebooting.

SD card booting works without any issues...

I noticed someone posted a vid on Youtube with the same issue, he has a normal harddrive connected to his Pi but with a separate power switch for his drive:

https://www.youtube.com/watch?v=XLyQY1J85-c

What can i do to resolve this problem??? Please help me out...

My system specs:

Rpi 4 4Gb Rpi OS 32-bit TP-LINK UH720 powered USB hub Orico-bh100 external SDD

Can't share logs because... Can't boot into system. :shock:

Thanks in advance!!! :(

Mind you this is an external SSD not an external SSD adapter...

Here's a link to the thread i opened on the official forums:

https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=295119

carras commented 3 years ago

@THX1180 The solution is in the first comment and in this comment: https://github.com/raspberrypi/rpi-eeprom/issues/266#issuecomment-746890101

THX1180 commented 3 years ago

@THX1180 The solution is in the first comment and in this comment: #266 (comment)

Yeah it's not working for me... I have to plug/unplug multiple times, the SSD boots when it wants to.

20201217_173433 20201217_173433

What is causing this???

carras commented 3 years ago

@THX1180 do the solution and try with the disk attach with no hub.

lurch commented 3 years ago

@carras Just because your solution works for you, I don't think it's safe to assume that it'll always work for everyone else? :shrug: (If it was really "one size fits all", I assume @timg236 would have already been doing that in the EEPROM code!)

THX1180 commented 3 years ago

@THX1180 do the solution and try with the disk attach with no hub.

Yeah that doesn't work... Same problem.

carras commented 3 years ago

@lurch give him something better. Just trying to help with what I know.

@THX1180 your case has the same controller, if I recommend to star over. Make sure your confi file keep with the custom config after reboots. Can you copy here your config file after a reboot?

THX1180 commented 3 years ago

@lurch give him something better. Just trying to help with what I know.

@THX1180 your case has the same controller, if I recommend to star over. Make sure your confi file keep with the custom config after reboots. Can you copy here your config file after a reboot?

Thanks for helping me... Do you mean the eeprom config file???

I already did a fresh install of everything, and applied the eeprom config fix but no change.

carras commented 3 years ago

@THX1180 Yes, when you do the command of the solution there is a kind of text file. After a reboot copy here the text in there.

THX1180 commented 3 years ago

@THX1180 Yes, when you do the command of the solution there is a kind of text file. After a reboot copy here the text in there.

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
USB_MSD_PWR_OFF_TIME=0

Maybe i need to change the values of the USB_MSD_PWR_OFF_TIME=0 ???

carras commented 3 years ago

@THX1180 I think you don't have the latest firmware. Use te command vcgencmd bootloader_version and you want this version date: Dec 11 2020

And my config is:

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
HDMI_DELAY=0
USB_MSD_PWR_OFF_TIME=0

Also check this comment, cuz it's maybe a different approach: https://github.com/home-assistant/operating-system/issues/1045#issuecomment-747592332

But I think if you get the latest EEPROM and USB_MSD_PWR_OFF_TIME=0 you will solve the problem. But idk with your JMS

You can also try changing values of USB_MSD_PWR_OFF_TIME=0. I've tried 5000, 1500, 1000, 500 and 0. And only 0 works for me. Make sure you reboot before any change and then "sudo shutdown now" (idk if there is any different, but I do that) then turn off power connect the SSD enclosure, remove SD card and connect the power.

THX1180 commented 3 years ago

@THX1180 I think you don't have the latest firmware. Use te command vcgencmd bootloader_version and you want this version date: Dec 11 2020

And my config is:

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
HDMI_DELAY=0
USB_MSD_PWR_OFF_TIME=0

Also check this comment, cuz it's maybe a different approach: home-assistant/operating-system#1045 (comment)

But I think if you get the latest EEPROM and USB_MSD_PWR_OFF_TIME=0 you will solve the problem. But idk with your JMS

You can also try changing values of USB_MSD_PWR_OFF_TIME=0. I've tried 5000, 1500, 1000, 500 and 0. And only 0 works for me. Make sure you reboot before any change and then "sudo shutdown now" (idk if there is any different, but I do that) then turn off power connect the SSD enclosure, remove SD card and connect the power.

Thank you carras... I will try your suggestions and report back!!!

Thank you for helping me out...šŸ‘šŸ‘šŸ‘

Update:

I figured it out!!!! It's a weird problem actually...

Updated the bootloader.

I tried all settings and the best setting was to leave the conf blank, but that's actually not the reason why it's not booting i think... When i physically pull the power plug of the usb-hub, and wait for about 40 seconds it will boot up just fine...

There's also a power button the hub, when i power down the hub i have to wait longer to boot up again.

When i turn it on too fast it will not work... Or sporadically...

This isn't normal behaviour from what i know...

What could be the issue???

timg236 commented 3 years ago

Managed to get an Orico (JMicron) NVMe SSD and an Elutang (RealTek). As expected the RealTek works and the JMicron seems to be fussy about the software-controlled USB port power off which is why USB_MSD_PWR_OFF_TIME=0 works.

For the moment the September 3rd bootloader (default in RPI Imager Rescue) seems to work or adding USB_MSD_PWR_OFF_TIME=0. Once the JMicron behaviour is fully understood then hopefully the generic change can be made but that won't be for a few weeks.

N.B. This shouldn't be an issue on the 8GB which has different power sequencing at boot.

THX1180 commented 3 years ago

Managed to get an Orico (JMicron) NVMe SSD and an Elutang (RealTek). As expected the RealTek works and the JMicron seems to be fussy about the software-controlled USB port power off which is why USB_MSD_PWR_OFF_TIME=0 works.

For the moment the September 3rd bootloader (default in RPI Imager Rescue) seems to work or adding USB_MSD_PWR_OFF_TIME=0. Once the JMicron behaviour is fully understood then hopefully the generic change can be made but that won't be for a few weeks.

N.B. This shouldn't be an issue on the 8GB which has different power sequencing at boot.

Really awesome to hear this news... So it has to do with power sequencing? How come it works instantaneously when pulling the plug, and leaving it out for an x amount seconds when replugging? (Mind you I am talking about when using a powered hub)...

Same thing when I use the button on the hub, and don't power on/off/on too fast. It's like something is kept in memory, or power needs to drain completely after it's turned off...

Could you test with the Jmicron and a powered hub? Boot/reboot turn on and off x amount of times..

Cheers!