Boot issues with RPi5 when moving/copying SD-cards between different units

[jofemodo]

Describe the bug

I have 4 x RBPi5 over my desk, let's name them U1 to U4. U1 is my working unit and the rest are for testing. When moving a working SD-card from unit U1 to another one, most of times it doesn't boot. When re-plug the SD-card in U1 again, it works. The booting process breaks at the very beginning and the LED lights solid green. I got some info then booting without SD-card, waiting the extended info and then plug the SD-card. Then i can see this:

And after this, nothing. Silence.

If i re-plug the SD-card in the U1 unit, it always work. If i re-plug the SD-card in any RPi4, it always work.

I've tried all kind of combinations but i can't reproduce the error in a consistent way (yet). I wanted to open this issue in the rpi-eeprom repo, but it has closed the issue tracker, so i open here and perhaps someone can give me some guidance or explain if this "armstub8-2712.bin" file is related with the issue. I can't find this file anywhere.

FYI, the SD-image is based in the latest RaspberryPiOs, updated to the last. The customization is nothing special:

• Deb packages
• Some custom C libraries and python code.
• Some systemd service to run specific software at startup
• Some config.txt and cmdline.txt adjustments, but when restoring the original from the vanilla OS, it keeps failing

Steps to reproduce the behaviour

• Burn latest RPi5
• Boot in U1 => It works
• Boot in U2-U4 => It works
• U1 -> apt-upgrade => It works in all units yet
• Customize, installing deb packages and some specific software (zynthian software)
• Power-off and move SD-card to other unit (U2-U4) => It doesn't boot! (see screen capture above)
• Boot in U1 again => It boots normally
• Boot in any Pi4 => It boots normally

Device (s)

Raspberry Pi 5

System

• Raspberry Pi OS Bookworm aarch64 (latest version, updated but NO rpi-update)
• Any firmware/bootloader. I tested to recover

Logs

See screen captures above. I could prepare a minimal customizing script to reproduce the issue.

Additional context

No response

[popcornmix]

I wanted to open this issue in the rpi-eeprom repo, but it has closed the issue tracker

Not sure what you mean - it has many open issues. I'll transfer it as this is not a linux issue (the linux kernel hasn't even been loaded at this point).

[nbuchwitz]

Afaik the bootloaders hands over to the Linux kernel in armstub ...

• Do you have an UART adapter at hand and could provide the output the bootloader console (more verbose)?
• How does you /boot/firmware/config.txt look like? Afaik zynthian uses some custom overlays

Edit: In https://github.com/zynthian/zynthian-sys/commit/b02222081b4427a5343aca2844337edee2db0727#diff-b0bb583397b992204dcb3c7952e87bf9640bcc35957191cb7fcf0f96c28d20e2 you pin the firmware to a certain release. Which version and what were the reasons?

[jofemodo]

Hi @nbuchwitz!

Thanks for your support.

No matter the firmware version, it always fails. I've tried the recovery image firmware, recent ones, etc. I observe the same behavior no matter the firmware version i use.

I've also tried to restore the original "vanilla" config.txt and cmdline.txt from a clean RaspberryPiOS image and the issue persists.

I can't find this "armstub8-2712.bin" file anywhere. Where is it supposed to be? In the boot/firmware directory?

I'm looking to boot with an UART cable to get more info about the boot process.

Thanks

[popcornmix]

There shouldn't be a armstub8-2712.bin - the bootload contains the default one embedded. If that file exists, then it will be used in preference, to allow custom stubs, but it's not present on a default RpiOS image.

[jofemodo]

OK! Understood. I think i've found the issue. It's my fault because i was not changing the root partition UUID when restoring the vanilla cmdline.txt. Now it boots with the vanilla config, so the issue is probably some problem with my custom config.txt.

I will close.

Thanks a lot for your support!