2022-07-22 firmware broke booting Ubuntu 22.04

[KcrPL]

Describe the bug

This morning I used the rpi-eeprom-update tool to update the RPi's EEPROM to 2022-07-22. While the upgrade worked, booting to the OS on the SD Card no longer did. All I had was the ACTivity light (green LED) blinking the error code. The blinking repeats.

4 long and 4 short ones. I couldn't really find a lot of info about it but after 3 hours I gave up and reverted to 2022-04-26 (that I was using before) and it's working now.

Steps to reproduce the behaviour

• Upgrade to 2022-07-22 firmware using the rpi-eeprom-update tool (Recovery also works)
• You see the Activity LED giving you the error code: 4 long blinks and 4 short ones. It repeats.

Device (s)

Raspberry Pi 4 Mod. B

Bootloader configuration.

# version initially shipped with bionic) don't understand the conditional
# [sections] below and simply ignore them. The Pi4 doesn't boot at all with
# firmwares this old so it's safe to place at the top. Of the Pi2 and Pi3, the
# Pi3 uboot happens to work happily on the Pi2, so it needs to go at the bottom
# to support old firmwares.

[pi4]
# commented by do-release-upgrade (LP: #1936401)
#kernel=uboot_rpi_4.bin
max_framebuffers=2

[pi2]
# commented by do-release-upgrade (LP: #1936401)
#kernel=uboot_rpi_2.bin

[pi3]
# commented by do-release-upgrade (LP: #1936401)
#kernel=uboot_rpi_3.bin

[all]
# added by do-release-upgrade (LP: #1936401)
kernel=vmlinuz
initramfs initrd.img followkernel
arm_64bit=1
# commented by do-release-upgrade (LP: #1936401)
#device_tree_address=0x03000000

# The following settings are "defaults" expected to be overridden by the
# included configuration. The only reason they are included is, again, to
# support old firmwares which don't understand the "include" command.

enable_uart=1
cmdline=cmdline.txt

# merged from syscfg.txt by do-release-upgrade (LP: #1936401)

enable_uart=1
dtparam=audio=on
dtparam=i2c_arm=on
dtparam=spi=on
cmdline=cmdline.txt
# merged from usercfg.txt by do-release-upgrade (LP: #1936401)

System

No response

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[lurch]

4 long and 4 short ones. I couldn't really find a lot of info about it

https://www.raspberrypi.com/documentation/computers/configuration.html#led-warning-flash-codes Although "Unsupported board type" seems unlikely to be the actual error, if an earlier bootloader-version works.

[KcrPL]

Unless I might be mistaken and it was 5 short ones.

[pelwell]

If you attach an HDMI monitor to it you should see diagnostic messages, at least up until start4.elf is entered.

[KcrPL]

Already tried this, without the SD Card plugged in it shows the diagnostic messages, when I plug it in, it's detected and immediately goes to a black screen and starts blinking the error code.

[pelwell]

You can plug the card in after the display wakes up. I think you'll find that it does load and execute start4.elf.

[KcrPL]

That's what I did. The result is the same.

[lurch]

Is the start4.elf in Ubuntu up-to-date? I've not used Ubuntu on a Pi, but I wonder if you could flash the 'older' boorloader to your eeprom, upgrade the packages inside Ubuntu to get a newer start4.elf; and maybe it then would boot with the newer bootloader? :shrug: But that's a bit of a random guess :wink:

[pelwell]

That suggests that the EEPROM is successfully launching the firmware, which makes reverting the EEPROM as a fix hard to explain.

[KcrPL]

Is the start4.elf in Ubuntu up-to-date? I've not used Ubuntu on a Pi, but I wonder if you could flash the 'older' boorloader to your firmware, upgrade the packages inside Ubuntu to get a newer start4.elf; and maybe it then would boot with the newer bootloader? 🤷 But that's a bit of a random guess 😉

md5sum /boot/start4.elf 7240e07245780edce65811faa9590d5c /boot/start4.elf

No idea how I can check the "version" of the file itself but here's the MD5 that you can compare with the latest.

[pelwell]

Without the SD card in, could you take a few photos of the diagnostic screens?

[pelwell]

strings start4.elf | grep VC_BUILD

[KcrPL]

strings start4.elf | grep VC_BUILD

I don't have time to upgrade it again and tinker with it now but it wasn't anything out of ordinary. It was just saying no SD Card found and then trying to boot off of USB then retrying. Here's the output.

VC_BUILD_ID_USER: dom
VC_BUILD_ID_TIME: 13:56:48
VC_BUILD_ID_VARIANT: start
VC_BUILD_ID_TIME: Jan 20 2022
VC_BUILD_ID_BRANCH: bcm2711_2
VC_BUILD_ID_HOSTNAME: buildbot
VC_BUILD_ID_PLATFORM: raspberrypi_linux
VC_BUILD_ID_VERSION: bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean)

[pelwell]

I think what is happening here is that the new EEPROM image is deciding that the version of start4.elf that you have is not new enough for the board that you have. Obviously it is booting for you with the version that you have, so it can't be very wrong.

It's almost certainly the case that if you were to upgrade the firmware on your SD card then the latest EEPROM image would be happy. If you have a spare SD card, try installing a current RPiOS image, upgrading it with sudo rpi-update, and upgrading the EEPROM. There is a slight chance that something is tricking the new EEPROM into thinking the board has a feature that it doesn't, and this test would confirm that when Ubuntu ship a newer version of the firmware that you will be able to update the EEPROM successfully.

[KcrPL]

Should I report this to Ubuntu?

[KcrPL]

Just updated the start4.elf on RPi running the firmware from 2022-04-26 and I'm facing the same issue. Also I can confirm it's 4 long flashes followed by 4 short flashes :----)

[pelwell]

No We don't know enough about the failure, and there is nothing they can do - they are shipping the same version of the firmware as we are.

[KcrPL]

That is true, I just upgraded to 2022-07-22 and I injected the latest start4.elf to no avail. The same error code is still there.

[KcrPL]

Also a question, was anyone able to reproduce the issue?

[lurch]

was anyone able to reproduce the issue?

It's the weekend...

[KcrPL]

Ah, right. Haha, sorry I lost track of time lol

[timg236]

Could be a file system bug. If you are able to use dd to capture a dump of the boot partition then we could take a look at that tomorrow

[timg236]

BOOT_UART=1 traces would also confirm this if you have a usb serial

[KcrPL]

Could be a file system bug. If you are able to use dd to capture a dump of the boot partition then we could take a look at that tomorrow

No idea how that could be an issue if it works on the April firmware and not on the latest one. Yesterday I also replaced everything in /boot with the files I got from fresh /boot Ubuntu 22.04 install but it didn't help.

BOOT_UART=1 traces would also confirm this if you have a usb serial

Unfortunately I don't have USB Serial so I can't check UART.

[pelwell]

It is possible to use a second Pi as a UART, if you have one.

[KcrPL]

Unfortunately I don't have a second Pi. Looking at the supply outages I won't be getting another one soon lol.

[KcrPL]

Should I try checking all BETA versions so we can pinpoint what was the last that was working?

[pelwell]

Understood. We should get a chance to run some tests after the weekend, at which point we'll know if this issue is in some way unique to your system.

[KcrPL]

Thank you. That's enough for today, enjoy your Sunday guys :----)

Let me know if I can be of any help.

[timg236]

Details of the SD card so that we can buy one would be good

[KcrPL]

I'm using a 16GB SanDisk Edge microSD Card.

[lurch]

What revision code does your Raspberry Pi have?

[KcrPL]

What revision code does your Raspberry Pi have?

Hardware        : BCM2835
Revision        : b03114
Serial          : 1000000065c99932
Model           : Raspberry Pi 4 Model B Rev 1.4

[waveform80]

Interesting; we're not yet shipping the July 22nd EEPROM in our version of the rpi-eeprom package (haven't had the time to bump it this cycle yet) but this could certainly occur if someone's switching OS'. I'll try and replicate here with the same board rev.

[timg236]

Edit: Tweaked title, we'd prefer to manage bug categories ourselves :)

The Raspberry Pi APT package hasn't been updated yet either.

The SD card is the same as our stock version in the office and it seems to be working here on Raspberry Pi. I haven't tried Ubuntu yet but unless it's a really obscure LFN issue I think it's unlikely to be OS specific. Possibly the board / card failed at 50MHz and also failed at 12.5 MHz but I've not seen that on otherwise good cards

[KcrPL]

Hah, name the title however it fits you guys :)

Interesting; we're not yet shipping the July 22nd EEPROM in our version of the rpi-eeprom package (haven't had the time to bump it this cycle yet) but this could certainly occur if someone's switching OS'. I'll try and replicate here with the same board rev.

I guess now it would be worth mentioning that I rely on this repo and not on Ubuntu updating the rpi-eeprom-package. I made this little script.

#!/bin/bash

cd rpi-eeprom
git pull

sudo cp -n firmware/critical/* /lib/firmware/raspberrypi/bootloader/critical

export BOOTFS=/boot/firmware

sudo ./rpi-eeprom-update

echo ""
echo "Apply update? [Y/N]"

read input
if [[ $input == "Y" || $input == "y" ]]; then
        sudo ./rpi-eeprom-update -a -d
else
        exit
fi

(I was using stable channel before but now I switched it to critical for obvious reasons :----))

[waveform80]

The Raspberry Pi APT package hasn't been updated yet either.

Yes, just noticed that -- still, looks like we're a few months behind in the devel release, and the LFN issue looks worth backporting to the LTS so I'll see if I can get that bumped soon.

The SD card is the same as our stock version in the office and it seems to be working here on Raspberry Pi. I haven't tried Ubuntu yet but unless it's a really obscure LFN issue I think it's unlikely to be OS specific. Possibly the board / card failed at 50MHz and also failed at 12.5 MHz but I've not seen that on otherwise good cards

Attempted replication on a 4GB rev 1.4 (I don't have a 2GB rev 1.4 unfortunately, just a rev 1.1). Flashed the EEPROM to the 2022-07-22 release under RaspiOS Bullseye, rebooted that successfully, then switched cards to an Ubuntu 22.04 card and booted that successfully, so I've been unable to replicate here, at least when booting from an SD card.

I wonder if there's anything else unusual about the OP's setup (beyond customizing which EEPROM is installed). @KcrPL could you run the following to verify which version of the bootloader firmware (not the EEPROM firmware, but start.elf and fixup.dat) you have installed, and what the checksums are for those files on your boot partition?

$ apt policy linux-firmware-raspi
linux-firmware-raspi:
  Installed: 6-0ubuntu3
  Candidate: 6-0ubuntu3
  Version table:
 *** 6-0ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports jammy/restricted arm64 Packages
        100 /var/lib/dpkg/status

$ sha256sum /boot/firmware/*.{elf,dat}
eeb7655f3d232f5708f30026b662fa7a450b5adbf6284b85ed07d4f6b69d0fd2  /boot/firmware/start.elf
1c9b93accc1deada2dcf2e4a92b96ad9024e3806c495afda419cd9352cc6442a  /boot/firmware/start4.elf
219d43494388d8c960e49e577417c412a039f6767320eda9fca0ec62f472340d  /boot/firmware/start4cd.elf
cd6b8338152195f40664f0b8e52792d35400924e50f837695f5141fa016f73e9  /boot/firmware/start4db.elf
305123602d919672ca6fcd8ae2dc7cf6b90ca165250dafec66acbf464482c7d9  /boot/firmware/start4x.elf
90f687a40b3ec2f2c73d6f5543047492b8b66a6eb236586cb9ba49037e2c1fb4  /boot/firmware/start_cd.elf
899a465d6768810955127216dab5b2c104dbea4efdc18962136685deb9551f0b  /boot/firmware/start_db.elf
7d1dcb7dc042b3097570d475b7f9145b812b06a4ca82b16106c23f4e4c8a2b5a  /boot/firmware/start_x.elf
406acd0484e6bbd333c6f1f1be2d5c4fe5855a1f6f1511a4550c7b034654db47  /boot/firmware/fixup.dat
ce9c0be09a40d74adc68bcd62ce8569f112369cb02f2d6f5b047c29c04535d66  /boot/firmware/fixup4.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  /boot/firmware/fixup4cd.dat
46a47ace2de856903ccae26725eeb09771b581fd724d6d2b68bbf39e7d5d975a  /boot/firmware/fixup4db.dat
0658d6c297f54ed415e5fc09b689d6f75f6b89839f605a6cb965e3ab99a38d78  /boot/firmware/fixup4x.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  /boot/firmware/fixup_cd.dat
08a79163d2f2e778b8f4d63a849e9220795a1cb985ea931761d298dbad9195f4  /boot/firmware/fixup_db.dat
a97569738067d3a2c11b92ff9069547d0f708645e12a547cc46ce49ec2baf128  /boot/firmware/fixup_x.dat

Given you're working on a system that's been through an upgrade to 22.04 (based on the comments in your config.txt), and given that we renamed linux-firmware-raspi2 to linux-firmware-raspi during that upgrade (which sounds trivial, but was horrifically complex in practice because of the dpkg-diversions in that package), I just want to confirm you've got the correct version installed, and that I haven't messed up flash-kernel so that the bootloader firmware is actually reaching the correct place!

[KcrPL]

That is correct. It was originally on Ubuntu 20.04 and I upgraded to 22.04.

kcrpl@raspberrypi-4-kcrpl:~$ apt policy linux-firmware-raspi
linux-firmware-raspi:
  Installed: 6-0ubuntu3
  Candidate: 6-0ubuntu3
  Version table:
 *** 6-0ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports jammy/restricted arm64 Packages
        100 /var/lib/dpkg/status

kcrpl@raspberrypi-4-kcrpl:~$ sha256sum /boot/firmware/*.{elf,dat}
eeb7655f3d232f5708f30026b662fa7a450b5adbf6284b85ed07d4f6b69d0fd2  /boot/firmware/start.elf
1c9b93accc1deada2dcf2e4a92b96ad9024e3806c495afda419cd9352cc6442a  /boot/firmware/start4.elf
219d43494388d8c960e49e577417c412a039f6767320eda9fca0ec62f472340d  /boot/firmware/start4cd.elf
cd6b8338152195f40664f0b8e52792d35400924e50f837695f5141fa016f73e9  /boot/firmware/start4db.elf
305123602d919672ca6fcd8ae2dc7cf6b90ca165250dafec66acbf464482c7d9  /boot/firmware/start4x.elf
90f687a40b3ec2f2c73d6f5543047492b8b66a6eb236586cb9ba49037e2c1fb4  /boot/firmware/start_cd.elf
899a465d6768810955127216dab5b2c104dbea4efdc18962136685deb9551f0b  /boot/firmware/start_db.elf
7d1dcb7dc042b3097570d475b7f9145b812b06a4ca82b16106c23f4e4c8a2b5a  /boot/firmware/start_x.elf
406acd0484e6bbd333c6f1f1be2d5c4fe5855a1f6f1511a4550c7b034654db47  /boot/firmware/fixup.dat
ce9c0be09a40d74adc68bcd62ce8569f112369cb02f2d6f5b047c29c04535d66  /boot/firmware/fixup4.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  /boot/firmware/fixup4cd.dat
46a47ace2de856903ccae26725eeb09771b581fd724d6d2b68bbf39e7d5d975a  /boot/firmware/fixup4db.dat
0658d6c297f54ed415e5fc09b689d6f75f6b89839f605a6cb965e3ab99a38d78  /boot/firmware/fixup4x.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  /boot/firmware/fixup_cd.dat
08a79163d2f2e778b8f4d63a849e9220795a1cb985ea931761d298dbad9195f4  /boot/firmware/fixup_db.dat
a97569738067d3a2c11b92ff9069547d0f708645e12a547cc46ce49ec2baf128  /boot/firmware/fixup_x.dat

I also mentioned before in the thread that I replaced the files in /boot with the new files straight from the Ubuntu 22.04 image but I kept the backup of the original files I had, here's the checksum of them.

$ sha256sum *.{elf,dat}
eeb7655f3d232f5708f30026b662fa7a450b5adbf6284b85ed07d4f6b69d0fd2  start.elf
1c9b93accc1deada2dcf2e4a92b96ad9024e3806c495afda419cd9352cc6442a  start4.elf
219d43494388d8c960e49e577417c412a039f6767320eda9fca0ec62f472340d  start4cd.elf
cd6b8338152195f40664f0b8e52792d35400924e50f837695f5141fa016f73e9  start4db.elf
305123602d919672ca6fcd8ae2dc7cf6b90ca165250dafec66acbf464482c7d9  start4x.elf
90f687a40b3ec2f2c73d6f5543047492b8b66a6eb236586cb9ba49037e2c1fb4  start_cd.elf
899a465d6768810955127216dab5b2c104dbea4efdc18962136685deb9551f0b  start_db.elf
7d1dcb7dc042b3097570d475b7f9145b812b06a4ca82b16106c23f4e4c8a2b5a  start_x.elf
406acd0484e6bbd333c6f1f1be2d5c4fe5855a1f6f1511a4550c7b034654db47  fixup.dat
ce9c0be09a40d74adc68bcd62ce8569f112369cb02f2d6f5b047c29c04535d66  fixup4.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  fixup4cd.dat
46a47ace2de856903ccae26725eeb09771b581fd724d6d2b68bbf39e7d5d975a  fixup4db.dat
0658d6c297f54ed415e5fc09b689d6f75f6b89839f605a6cb965e3ab99a38d78  fixup4x.dat
9e94104b9a58210f9c0250cbc567da164f5a1fb6930b57e305b78f72a2e22799  fixup_cd.dat
08a79163d2f2e778b8f4d63a849e9220795a1cb985ea931761d298dbad9195f4  fixup_db.dat
a97569738067d3a2c11b92ff9069547d0f708645e12a547cc46ce49ec2baf128  fixup_x.dat

[timg236]

Please could you try leaving the HDMI monitor connected with it flashing in the error state and making sure that you didn't disable HDMI in the bootloader-config. Without the error message on HDMI or the UART log it will be very difficult to help you unless we can reproduce the error

[KcrPL]

HDMI is not disabled. When there's no SD Card plugged in I can see the bootloader screen just fine, when I plug it in, it starts booting from it (no error messages etc.), HDMI literally turns off (monitor says no signal) and I get the blinking error code.

There's no other way to read UART log without USB Serial, right?

[timg236]

If you have a PC / other Pi and are familiar with WireShark then it's possible to use NETCONSOLE to get the bootloader logs. It just writes the log data to UDP packets to a broadcast address.

https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#NETCONSOLE

[KcrPL]

Very well. I will try it later.

[...] although DHCP is not attempted unless network boot is requested. - I have to configure networking manually in config.txt , right?

[timg236]

You have to add the NETCONSOLE line to the EEPROM config. Since it just writes packets to the network it doesn't need any network configuration. I normally connect the cable directly to the host running Wireshark and capture all packets

[waveform80]

Ah ha! I may have replicated this, but it's not a precise replication and I do think this is subject to FS layout. Here's what I'm observing:

Hardware: Pi 4 4GB rev 1.4 (revision c03114) Software: RaspiOS Lite arm64 (specifically the image https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-04-07/2022-04-04-raspios-bullseye-arm64-lite.img.xz -- the current image served by rpi-imager) Software: Ubuntu 22.04 LTS for arm64 (specifically the image at http://cdimage.ubuntu.com/releases/jammy/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz -- this is also currently served by rpi-imager)

With the EEPROM release from 2022-04-26 installed, this boots both the RaspiOS Lite image, and the Ubuntu 22.04 LTS image successfully. With the EEPROM release from 2022-07-22 installed, the RaspiOS Lite image boots successfully, but the Ubuntu 22.04 LTS image does not.

This same set up is the one I failed at replicating the problem with earlier, but notably that card was an Ubuntu 22.04 image that I'd already upgraded with a new kernel (amongst other things) so the boot partition was certainly not the same as the release image.

Having said that, I'm not observing the HDMI turning black as @KcrPL is, nor am I observing 4 long and 4 short pulses. Instead I'm getting 4 short pulses, while the diagnostics screen complains that it cannot load the firmware from the boot partition of the SD card. Hence, I'm not certain this is exactly the same issue -- but I have reverted the EEPROM, successfully booted the (freshly flashed) Ubuntu 22.04 image, then updated back to EEPROM 2022-07-22 and failed to boot (another freshly flashed) Ubuntu 22.04 image twice now so there's a problem here at least. I'm happy to open a separate ticket if that's desirable. My next step is to dig out serial and see if I can get a useful log from the failing boot.

[waveform80]

Sorry this took a bit longer than expected! Had to deal with my not noticing that, when editing the boot config to include BOOT_UART=1 in the EEPROM config, I'd inadvertently downgraded the EEPROM back to the current stable and then spent a while scratching my head going "why is it working now?!".

Anyway, here's the UART output from attempting (and failing) to boot Ubuntu 22.04 on EEPROM version 2022-07-22 on my rev 1.4:

RPi: BOOTLOADER release VERSION:4fde6470 DATE: 2022/07/22 TIME: 13:37:49 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1658493469 0x1fc41cb1 0x00c03114 0x0006cc82
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x1 total-size: 32 Gbit 3200
DDR 3200 0 0 32 152

XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 11
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order f4
USB2[1] 400202e1 connected
USB2 root HUB port 1 init
DEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431
HUB init [01:00] 2.16 000000:01
HUB [01:00] 2.16 000000:01 init port 4 speed 3
DEV [02:01] 2.00 000004:01 class 9 VID 058f PID 6254
HUB init [02:01] 2.00 000004:01
HUB [02:01] 2.00 000004:01 init port 4 speed 2
DEV [03:02] 2.00 000044:01 class 0 VID 04d9 PID a088
HID [03:02] 2.00 000044:01 register HID
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8000 [13]
CID: 00035344534333324780e456b680013c
CSD: 400e00325b590000edc87f800a404000
SD: bus-width: 4 spec: 2 SCR: 0x02358043 0x00000000
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x00000800,  524288 type: 0x0c
MBR: 0x00080800, 7129360 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 2048 oem: 'mkfs.fat' volume: ' system-boot'
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Trying partition: 0
type: 32 lba: 2048 oem: 'mkfs.fat' volume: ' system-boot'
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Firmware not found
ERROR: 4
Retry SD 1
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8000 [13]
CID: 00035344534333324780e456b680013c
CSD: 400e00325b590000edc87f800a404000
SD: bus-width: 4 spec: 2 SCR: 0x02358043 0x00000000
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x00000800,  524288 type: 0x0c
MBR: 0x00080800, 7129360 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 2048 oem: 'mkfs.fat' volume: ' system-boot'
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Trying partition: 0
type: 32 lba: 2048 oem: 'mkfs.fat' volume: ' system-boot'
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Firmware not found
ERROR: 4
Boot mode: USB-MSD (04) order f
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 18
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 19
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
USB2[1] 400202e1 connected
USB2 root HUB port 1 init
DEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431
HUB init [01:00] 2.16 000000:01
HUB [01:00] 2.16 000000:01 init port 4 speed 3
DEV [02:01] 2.00 000004:01 class 9 VID 058f PID 6254
HUB init [02:01] 2.00 000004:01
HUB [02:01] 2.00 000004:01 init port 4 speed 2
DEV [03:02] 2.00 000044:01 class 0 VID 04d9 PID a088
HID [03:02] 2.00 000044:01 register HID
HDMI0 edid block 0 offset 0
00ffffffffffff0009d1db7845540000
071a010380301b782ef003a65552a027
115053a56b80d1c081c081008180a9c0
b30001010101023a801871382d40582c
4500dc0c1100001e000000ff00463247
30313035383031390a20000000fd0032
4c1e5311000a202020202020000000fc
0042656e51204757323237300a2001c4
HDMI0 edid block 1 offset 128
020322f14f901f051404130312071615
01061102230907078301000065030c00
2000023a801871382d40582c4500dc0c
1100001e011d8018711c1620582c2500
dc0c1100009e011d007251d01e206e28
5500dc0c1100001e8c0ad08a20e02d10
103e9600dc0c11000018000000000000
00000000000000000000000000000071
HDMI0: best-mode 1 (limit 1) 1280x720 60 Hz CEA modes fe007f80000000000000000000000000
USB MSD stopped. Timeout: 25 seconds
Boot mode: RESTART (0f) order 0
Restart 0 max -1

There's some more after that but it just goes round in circles not finding the boot firmware. For comparison, here's the output when (successfully) booting RaspiOS Lite with the same EEPROM on the same board:

RPi: BOOTLOADER release VERSION:4fde6470 DATE: 2022/07/22 TIME: 13:37:49 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1658493469 0x1fc41cb1 0x00c03114 0x0006d61c
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x1 total-size: 32 Gbit 3200
DDR 3200 0 0 32 152

XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 11
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order f4
USB2[1] 400202e1 connected
USB2 root HUB port 1 init
DEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431
HUB init [01:00] 2.16 000000:01
HUB [01:00] 2.16 000000:01 init port 4 speed 3
DEV [02:01] 2.00 000004:01 class 9 VID 058f PID 6254
HUB init [02:01] 2.00 000004:01
HUB [02:01] 2.00 000004:01 init port 4 speed 2
DEV [03:02] 2.00 000044:01 class 0 VID 04d9 PID a088
HID [03:02] 2.00 000044:01 register HID
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8000 [16]
CID: 000353445343313647809d3640ca011b
CSD: 400e00325b59000076b27f800a404000
SD: bus-width: 4 spec: 2 SCR: 0x02358443 0x00000000
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x00002000,  524288 type: 0x0c
MBR: 0x00082000,30583808 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Read config.txt bytes     2109 hnd 0x391
Read start4.elf bytes  2241824 hnd 0x58f2
Read fixup4.dat bytes     5352 hnd 0x3a6
0x00c03114 0x00000000 0x00000fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: e5a963efa66a1974127860b42e913d2374139ff5 Mar 24 2022 13:19:26
Starting start4.elf @ 0xfec00200 partition 0
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 18
+

If it's of any help, I'm happy to gather output from booting with other firmware versions, and/or other board revs (though the only other rev I have easy access to is board rev 1.1).

[timg236]

Looks like it might be a file-system regression. Would you be able so share a copy of the boot partition so that we can examine it?

sudo dd if=/dev/mmcblk0p1 of=boot.img bs=1M
gzip boot.img

[timg236]

Actually, I might have found something. Leave it with me

[waveform80]

Certainly -- though it may be quicker for you to grab for yourselves, as it's literally the Ubuntu 22.04 LTS for arm64 image from rpi-imager (which derives from http://cdimage.ubuntu.com/releases/jammy/release/ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz) so unpacking that image and nabbing the boot partition from it should probably give you what you need.

That said, I think rpi-imager is writing a customized cloud-init config to the boot partition so maybe that's causing the layout issue (though I'd be surprised, given I wouldn't have thought that would be messing with the blocks for the bootloader)?

Oh, just seen your other reply! No problem -- yell if you still need the boot partition and I'll stick it somewhere accessible.

[timg236]

I think this should fix it, there was a bug causing cached directory entries to be overwritten in the new LFN code

rpi-eeprom-recovery-issue438.zip

Edit: Was able to reproduce this with the 20.04 image, I think in Raspberry Pi OS there were fewer directory entries and the access pattern worked by luck. I'll need to tweak the FAT auto-test to interleave some more plausible file-names in-between real firmware filenames

[KcrPL]

Do you still need the image of /boot partition?

Also, will test the firmware and tell you if it's working now.

[KcrPL]

I think this should fix it, there was a bug causing cached directory entries to be overwritten in the new LFN code

rpi-eeprom-recovery-issue438.zip

Edit: Was able to reproduce this with the 20.04 image, I think in Raspberry Pi OS there were fewer directory entries and the access pattern worked by luck. I'll need to tweak the FAT auto-test to interleave some more plausible file-names in-between real firmware filenames

It's working! After flashing the new firmware it's working now! :)