Bug Report: Can't boot from NVME if USC C SSD is plugged in.

[munecito]

What happened?

Hello.

I have an Orange Pi 5 16Gb. I have installed Ubuntu server on the NVME drive using the step in this post:

https://github.com/Joshua-Riek/ubuntu-rockchip/issues/813#issuecomment-2119203993

It works great, the issue starts if I plug a Toshiba 500gb USB C portable drive and leave it plugged. If I do a sudo reboot the system restarts fine, but if I was to do a sudo shutdown and then turn on the board then it won't boot.

By looking at the LEDs in both the pi and the SSD I can only assume that it is trying to boot from the SSD as the Pi green LED blinks a couple of times, then the SSD blink and then the Pi green LED stops doing anything for a few seconds, and repeat the whole sequence.

It seems to me that it gets trapped in a boot loop.

I also tried to fix it by following the steps in this post:

https://github.com/Joshua-Riek/ubuntu-rockchip/issues/41

The folder orangepi did not exist so I created one and ran

wget https://github.com/Joshua-Riek/ubuntu-orange-pi5/files/10889255/rkspi_loader.zip

unzipped and then ran the command

sudo dd if=/usr/share/orangepi/rkspi_loader.img of=/dev/mtdblock0

IT still does the same. On start it gets trapped in a boot loop until I physically disconnect the USB SSD.

The output of fdisk -l is below:

$ sudo fdisk -l [sudo] password for will: Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/loop0: 42.88 MiB, 44961792 bytes, 87816 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop1: 92.03 MiB, 96497664 bytes, 188472 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop2: 33.71 MiB, 35344384 bytes, 69032 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop3: 69.21 MiB, 72572928 bytes, 141744 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop4: 59.77 MiB, 62676992 bytes, 122416 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop5: 69.24 MiB, 72605696 bytes, 141808 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop6: 94.54 MiB, 99127296 bytes, 193608 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop7: 94.43 MiB, 99016704 bytes, 193392 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mtdblock0: 16 MiB, 16777216 bytes, 32768 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 8AA72D9E-24E9-41C7-8EF2-D35FB3A900AC

Device Start End Sectors Size Type /dev/mtdblock0p1 64 7167 7104 3.5M Linux filesystem /dev/mtdblock0p2 7168 7679 512 256K Linux filesystem /dev/mtdblock0p3 7680 8063 384 192K Linux filesystem /dev/mtdblock0p4 8064 8127 64 32K Linux filesystem /dev/mtdblock0p5 8128 8191 64 32K Linux filesystem /dev/mtdblock0p6 8192 16383 8192 4M Linux filesystem /dev/mtdblock0p7 16384 32734 16351 8M Linux filesystem

Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: KBG50ZNS512G NVMe KIOXIA 512GB Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 04687F84-DDF8-4D7C-84DB-7A7DBC281114

Device Start End Sectors Size Type /dev/nvme0n1p1 32768 40959 8192 4M Microsoft basic data /dev/nvme0n1p2 40960 1000214527 1000173568 476.9G EFI System

Can someone please tell me why it fails to boot normally from the NVME drive?

Thanks.

Kernel version

6.1.0-1025-rockchip

SBC model

Orange Pi 5

What operating system are you seeing this problem on?

Ubuntu 24.04 LTS (Noble Nombat)

Relevant logs

No response

[ewaldc]

@munecito, I have seen this on my OPI5+ systems as well. I think this is related on how the default u-boot boot sequence works, very much like the default UEFI boot order on a PC. Generally USB and SD card come ahead of NVME and EMMC, which in a way is handy e.g. if one messes up the kernel on the NVME, one can push in the SD card and let the system boot from there as to recover the NVME boot.

Normally, if there is an obvious failure (e.g. USB stick not formatted), u-boot can avoid booting from this media and/or recover from a failed boot, and then move to the next boot sequence e.g. the NVME. But often u-boot can not gain back proper control after it attempted an erroneous boot e.g. boot an x86 kernel on an arm64 system. If it crashes or hangs (u-boot watchdog timer will catch this), it will reset the HW and then you end up with the endless (failing) boot loop.

I am very sure that you can change the default boot order using the proper u-boot commands and then save it to non-volatile memory, I just have not (yet) done it for these boards... Normally, you can use the commands fw_printenv and fw_setenv to find out the current boot sequence and modify it, but it is not working on my systems, probably because either the device tree does not include the proper flash memory zones and/or the proper flash memory drivers are not loaded. Another way is via an USB terminal connection, so you can interrupt the boot process and interact with u-boot (printenv command is a good starting point) directly. Finally you could insert changes via u-boot-update, but that is a quite risky process.

[munecito]

@ewaldc those commands don't work on my system either (fw_printenv and fw_setenv).

I don't have a way to see the boot order at the moment.

I would like to try to find a way to resolve or fix this issue as I intend to deploy the Orange Pi 5 running nextcloud at my parents house to backup my own instance remotely. The issue being that if the system was to loose power or have a hard reboot it won't be able to self recover without physical intervention.

I don't have much linux knowledge. So I wouldn't know where to start. Although I had already broken the kernel twice tinkering with files.

Is it a formatting issue with the SDD? Do you think I may not have the ideal filesystem in the portable SSD causing u-boot to fail to regain control?

Is there a u-boot log that I can look at to see and report what the computer sees?

Thanks.

[ewaldc]

@munecito, for your convenience I have attached a u-boot environment print. To control u-boot, the easiest way is to attach a console (gnd, tx, rx pins on the left side of the board via 3 wires to a USB serial board e.g. a CH340, never connect 5V, it will kill your board). Shortly after boot, hit CTRL C. Then you can issue printenv or any other command. When done: saveenv to make your change permanent in flash. There are other ways e.g. linux kernel netconsole but that is more for kernel dev folks :-)

I have not examined yet what exact things need to be changed to skip USB, but there is plenty of u-boot documentation. Maybe it is as simple as remove usb0 from boot_targets... scan_dev_for_... variables are definately another place to look at... Hope this can get you going. U-Boot 2017.09.txt

[munecito]

@ewaldc I will have a little wait to see if one of the devs pops up with an idea before I try making physical connections ;)

Thank you so much. I'll keep your printout to study it.

[Joshua-Riek]

Honestly, it's not worth reporting or looking into this problem unless it's with mainline U-Boot, but support for these devices is a joke. I'll be closing and locking this tread. I'll also likely be archiving this project, or deleting it outright. Fork while you can.