Recent RaspianOS image 09-06-2022 does not boot on CM4 from NVMe

[framps]

I just upgraded my RaspberyOS Bullseye on my CM4 with sudo update && sudo upgrade and finally wasn't able to boot the system any more :-(.

Then I dd'd the actual image on the eMMc of the CM4 and everything works fine. When I dd'd the image on NMVe the system doesn't boot.

The image I used and does not boot on NVMe is 2022-09-06-raspios-bullseye-armhf-lite.img.

When I restore a backup I have created some time ago the system boots from NVMe. The system also boots when I restore 2022-04-04-raspios-bullseye-armhf-lite.img which is the base of my restored backup.

Looks like there is some issue with NVMe on latest image?

[XECDesign]

@timg236 is this something you might know about?

[timg236]

Not really, but flashing latest RPi OS 32-bit on a CM4 NVMe works for me

[framps]

works for me

Thank you very much for your reply. Looks like there is something wrong on my CM4. How can I debug the issue?

[peterharperuk]

2022-09-06-raspios-bullseye-armhf-lite.img works for me, as does upgrading from 2022-04-04-raspios-bullseye-armhf-lite.img to 2022-09-06-raspios-bullseye-armhf-lite.img.

Can you give more details of your setup? Are you using a Raspberry PI compute module IO board? What's the make and model of your NVMe drive? Do you see anything on the screen when it fails?

You could edit /boot/cmdline.txt to remove the words "quiet splash". This will show the kernel booting on the screen if it's getting that far. If you have a UART cable I can tell you how to get logs from the bootloader and firmware.

For the avoidance of doubt you can run the following commands to give us some debug info...

rpi-eeprom-config cat /etc/rpi-issue vcgencmd bootloader_versionv vcgencmd version uname -a

sudo apt-get install nvme-cli sudo nvme list sudo nvme id-ctrl -H /dev/nvme0 sudo nvme list-ns /dev/nvme0 sudo nvme id-ns -H /dev/nvme0 --namespace-id=1

[timg236]

Adding enable_uart=1 and uart_2ndstage=1 to config.txt will also provide more information https://www.raspberrypi.com/documentation/computers/config_txt.html#uart_2ndstage https://www.raspberrypi.com/documentation/computers/config_txt.html#enable_uart

[framps]

Can you give more details of your setup?

It's a CM4 with 1GB RAM, 32 GB eMMC and 128 GB NVMe assembled on a Waveshare CM4-IO-BASE-A (See here for a picture of all components).

What's the make and model of your NVMe drive?

It's a hynix. See the linked picture and the nvme command results below.

Do you see anything on the screen when it fails?

Nothing. Green LEDs M.2 and ACT flash but the red boot LED is dark. Looks like it's just not booting.

So that's what I did: I copied the 09-06 image on eMMC and started the CM4 that way in order to be able to execute the following commands:

pi@cm4-bullseye:~ $ rpi-eeprom-config 
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0

# Boot Order Codes, from https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#BOOT_ORDER
# Try SD first (1), followed by, USB PCIe, NVMe PCIe, USB SoC XHCI then network
BOOT_ORDER=0xf25641
# Try NVMe first (6), followed by, USB PCIe, SD , USB SoC XHCI then network
#BOOT_ORDER=0xf25146

# Set to 0 to prevent bootloader updates from USB/Network boot
# For remote units EEPROM hardware write protection should be used.
ENABLE_SELF_UPDATE=1

Note: I use the second BOOT_ORDER when I want to boot from NVMe.

i@cm4-bullseye:~ $ cat /etc/rpi-issue 
Raspberry Pi reference 2022-09-06
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 66255495f29be5d09b765d081aff6fc0f11e59b4, stage2

i@cm4-bullseye:~ $ vcgencmd bootloader_version
2022/08/02 16:55:05
version 91b6280c53c0b6dff9e3d58810f439e577408845 (release)
timestamp 1659455705
update-time 1663329705
capabilities 0x0000007f

pi@cm4-bullseye:~ $ vcgencmd version
Aug 26 2022 14:03:33 
Copyright (c) 2012 Broadcom
version 102f1e848393c2112206fadffaaf86db04e98326 (clean) (release) (start_cd)

pi@cm4-bullseye:~ $ uname -a
Linux cm4-bullseye 5.15.61-v7l+ #1579 SMP Fri Aug 26 11:13:03 BST 2022 armv7l GNU/Linux

pi@cm4-bullseye:~ $ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1        FNABN43081010B54Z BC711 NVMe SK hynix 128GB                1         128.04  GB / 128.04  GB    512   B +  0 B   41002131
pi@cm4-bullseye:~ $ sudo nvme id-ctrl -H /dev/nvme0
NVME Identify Controller:
vid       : 0x1c5c
ssvid     : 0x1c5c
sn        :    FNABN43081010B54Z
mn        : BC711 NVMe SK hynix 128GB               
fr        : 41002131
rab       : 3
ieee      : ace42e
cmic      : 0
  [3:3] : 0 ANA not supported
  [2:2] : 0 PCI
  [1:1] : 0 Single Controller
  [0:0] : 0 Single Port

mdts      : 6
cntlid    : 0x1
ver       : 0x10300
rtd3r     : 0x7a120
rtd3e     : 0x1e8480
oaes      : 0x200
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
  [9:9] : 0x1   Firmware Activation Notices Supported
  [8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt    : 0x10
  [9:9] : 0 UUID List Not Supported
  [7:7] : 0 Namespace Granularity Not Supported
  [5:5] : 0 Predictable Latency Mode Not Supported
  [4:4] : 0x1   Endurance Groups Supported
  [3:3] : 0 Read Recovery Levels Not Supported
  [2:2] : 0 NVM Sets Not Supported
  [1:1] : 0 Non-Operational Power State Permissive Not Supported
  [0:0] : 0 128-bit Host Identifier Not Supported

rrls      : 0
cntrltype : 0
  [7:2] : 0 Reserved
  [1:0] : 0 Controller type not reported
fguid     : 
crdt1     : 0
crdt2     : 0
crdt3     : 0
oacs      : 0x17
  [9:9] : 0 Get LBA Status Capability Not Supported
  [8:8] : 0 Doorbell Buffer Config Not Supported
  [7:7] : 0 Virtualization Management Not Supported
  [6:6] : 0 NVMe-MI Send and Receive Not Supported
  [5:5] : 0 Directives Not Supported
  [4:4] : 0x1   Device Self-test Supported
  [3:3] : 0 NS Management and Attachment Not Supported
  [2:2] : 0x1   FW Commit and Download Supported
  [1:1] : 0x1   Format NVM Supported
  [0:0] : 0x1   Security Send and Receive Supported

acl       : 3
aerl      : 7
frmw      : 0x16
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x3   Number of Firmware Slots
  [0:0] : 0 Firmware Slot 1 Read/Write

lpa       : 0x1e
  [4:4] : 0x1   Persistent Event log Supported
  [3:3] : 0x1   Telemetry host/controller initiated log page Supported
  [2:2] : 0x1   Extended data for Get Log Page Supported
  [1:1] : 0x1   Command Effects Log Page Supported
  [0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe      : 255
npss      : 4
avscc     : 0x1
  [0:0] : 0x1   Admin Vendor Specific Commands uses NVMe Format

apsta     : 0x1
  [0:0] : 0x1   Autonomous Power State Transitions Supported

wctemp    : 356
cctemp    : 358
mtfa      : 0
hmpre     : 0
hmmin     : 0
tnvmcap   : 0
unvmcap   : 0
rpmbs     : 0
 [31:24]: 0 Access Size
 [23:16]: 0 Total Size
  [5:3] : 0 Authentication Method
  [2:0] : 0 Number of RPMB Units

edstt     : 8
dsto      : 1
fwug      : 0
kas       : 0
hctma     : 0x1
  [0:0] : 0x1   Host Controlled Thermal Management Supported

mntmt     : 273
mxtmt     : 355
sanicap   : 0x2
  [31:30] : 0   Additional media modification after sanitize operation completes successfully is not defined
  [29:29] : 0   No-Deallocate After Sanitize bit in Sanitize command Supported
    [2:2] : 0   Overwrite Sanitize Operation Not Supported
    [1:1] : 0x1 Block Erase Sanitize Operation Supported
    [0:0] : 0   Crypto Erase Sanitize Operation Not Supported

hmminds   : 0
hmmaxd    : 0
nsetidmax : 0
endgidmax : 1
anatt     : 0
anacap    : 0
  [7:7] : 0 Non-zero group ID Not Supported
  [6:6] : 0 Group ID does not change
  [4:4] : 0 ANA Change state Not Supported
  [3:3] : 0 ANA Persistent Loss state Not Supported
  [2:2] : 0 ANA Inaccessible state Not Supported
  [1:1] : 0 ANA Non-optimized state Not Supported
  [0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0
nanagrpid : 0
pels      : 1
sqes      : 0x66
  [7:4] : 0x6   Max SQ Entry Size (64)
  [3:0] : 0x6   Min SQ Entry Size (64)

cqes      : 0x44
  [7:4] : 0x4   Max CQ Entry Size (16)
  [3:0] : 0x4   Min CQ Entry Size (16)

maxcmd    : 0
nn        : 1
oncs      : 0x5f
  [7:7] : 0 Verify Not Supported
  [6:6] : 0x1   Timestamp Supported
  [5:5] : 0 Reservations Not Supported
  [4:4] : 0x1   Save and Select Supported
  [3:3] : 0x1   Write Zeroes Supported
  [2:2] : 0x1   Data Set Management Supported
  [1:1] : 0x1   Write Uncorrectable Supported
  [0:0] : 0x1   Compare Supported

fuses     : 0
  [0:0] : 0 Fused Compare and Write Not Supported

fna       : 0
  [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0 Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0 Format Applies to Single Namespace(s)

vwc       : 0x1
  [2:1] : 0 Support for the NSID field set to FFFFFFFFh is not indicated
  [0:0] : 0x1   Volatile Write Cache Present

awun      : 0
awupf     : 0
nvscc     : 1
  [0:0] : 0x1   NVM Vendor Specific Commands uses NVMe Format

nwpc      : 0
  [2:2] : 0 Permanent Write Protect Not Supported
  [1:1] : 0 Write Protect Until Power Supply Not Supported
  [0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu      : 0
sgls      : 0
 [1:0]  : 0 Scatter-Gather Lists Not Supported

mnan      : 0
subnqn    : nqn.2021-11.com.skhynix:nvme:nvm-subsystem-sn-FNABN43081010B54Z
ioccsz    : 0
iorcsz    : 0
icdoff    : 0
ctrattr   : 0
  [0:0] : 0 Dynamic Controller Model

msdbd     : 0
ps    0 : mp:6.3000W operational enlat:5 exlat:5 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:2.4000W operational enlat:30 exlat:30 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:1.9000W operational enlat:100 exlat:100 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0500W non-operational enlat:1000 exlat:1000 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0040W non-operational enlat:1000 exlat:9000 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
pi@cm4-bullseye:~ $ sudo nvme list-ns /dev/nvme0
[   0]:0x1
pi@cm4-bullseye:~ $ sudo nvme id-ns -H /dev/nvme0 --namespace-id=1
NVME Identify Namespace 1:
nsze    : 0xee7c2b0
ncap    : 0xee7c2b0
nuse    : 0xee7c2b0
nsfeat  : 0
  [4:4] : 0 NPWG, NPWA, NPDG, NPDA, and NOWS are Not Supported
  [2:2] : 0 Deallocated or Unwritten Logical Block error Not Supported
  [1:1] : 0 Namespace uses AWUN, AWUPF, and ACWU
  [0:0] : 0 Thin Provisioning Not Supported

nlbaf   : 1
flbas   : 0
  [4:4] : 0 Metadata Transferred in Separate Contiguous Buffer
  [3:0] : 0 Current LBA Format Selected

mc      : 0
  [1:1] : 0 Metadata Pointer Not Supported
  [0:0] : 0 Metadata as Part of Extended Data LBA Not Supported

dpc     : 0
  [4:4] : 0 Protection Information Transferred as Last 8 Bytes of Metadata Not Supported
  [3:3] : 0 Protection Information Transferred as First 8 Bytes of Metadata Not Supported
  [2:2] : 0 Protection Information Type 3 Not Supported
  [1:1] : 0 Protection Information Type 2 Not Supported
  [0:0] : 0 Protection Information Type 1 Not Supported

dps     : 0
  [3:3] : 0 Protection Information is Transferred as Last 8 Bytes of Metadata
  [2:0] : 0 Protection Information Disabled

nmic    : 0
  [0:0] : 0 Namespace Multipath Not Capable

rescap  : 0
  [6:6] : 0 Exclusive Access - All Registrants Not Supported
  [5:5] : 0 Write Exclusive - All Registrants Not Supported
  [4:4] : 0 Exclusive Access - Registrants Only Not Supported
  [3:3] : 0 Write Exclusive - Registrants Only Not Supported
  [2:2] : 0 Exclusive Access Not Supported
  [1:1] : 0 Write Exclusive Not Supported
  [0:0] : 0 Persist Through Power Loss Not Supported

fpi     : 0
  [7:7] : 0 Format Progress Indicator Not Supported

dlfeat  : 0
  [4:4] : 0 Guard Field of Deallocated Logical Blocks is set to 0xFFFF
  [3:3] : 0 Deallocate Bit in the Write Zeroes Command is Not Supported
  [2:0] : 0 Bytes Read From a Deallocated Logical Block and its Metadata are Not Reported

nawun   : 0
nawupf  : 0
nacwu   : 0
nabsn   : 0
nabo    : 0
nabspf  : 0
noiob   : 0
nvmcap  : 0
nsattr  : 0
nvmsetid: 0
anagrpid: 0
endgid  : 1
nguid   : 00000000000000000000000000000000
eui64   : ffffffffffffffff
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 

Adding enable_uart=1 and uart_2ndstage=1 to config.txt will also provide more information

I never used the serial to see messages from the CM4. Let me know if this information is important for you and I will connect my USB2TTL adapter to the CM4.

[peterharperuk]

It sounds like it's failing in the bootloader or firmware, in which case the uart output would be helpful. If you add enable_uart=1 and uart_2ndstage=1 to config.txt it'll tell us if it's getting as far as firmware.

If you can edit your eeprom config then setting BOOT_UART=1 will tell us if it fails in the bootloader.

[framps]

I just detected my USB2TTL Adapter is broken :-( .

I just ordered a new one and come back. But it will take a while until I can continue to work on this issue (approx 2 weeks).

[peterharperuk]

Ok. I've ordered a drive like yours (seemed suspiciously cheap) and will let you know if it works for me.

[framps]

I detected the ESP on my breadboard I used the adapter before seems to be broken instead of the USB2TTL adapter. I finally enabled uart first on the eMMC storage and got output. Then I enabled uart on the NVMe storage and also got output :-)

I'm sorry for the confusion but it's the first time I use uart on a Raspberry :-)

That's the output I get

RPi: BOOTLOADER release VERSION:91b6280c DATE: 2022/08/02 TIME: 16:55:05 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1659455705 0x34a87b72 0x00a03140 0x00069525
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 8Gb x1 total-size: 8 Gbit 3200
DDR 3200 0 0 8 152

Boot mode: NVME (06) order f2514
VID 0x1c5c MN BC711 NVMe SK hynix 128GB              
NVME on
MBR: 0x00002000,  524288 type: 0x0c
MBR: 0x00082000,249537200 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 1020 c-count 130554 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 130554
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 1020 c-count 130554 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 130554
Read config.txt bytes     2075 hnd 0x163
Read start4.elf bytes  2249280 hnd 0x3cdb
Read fixup4.dat bytes     5399 hnd 0x169
0x00a03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 102f1e848393c2112206fadffaaf86db04e98326 Aug 26 2022 14:03:16
Starting start4.elf @ 0xfec00200 partition 0
+

[timg236]

If you add uart_2ndstage=1 then you should get some more output from start4.elf

[framps]

It's already in there:

cat /media/framp/boot1/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=be6c5e24-02 rootfstype=ext4 fsck.repair=yes rootwait enable_uart=1 uart_2ndstage=1

[framps]

That's in config.txt:

cat /media/framp/boot1/config.txt 
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2

# Disable compensation for displays with overscan
disable_overscan=1

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[all]

[pi4]
# Run as fast as firmware / board allows
arm_boost=1

[all]
gpu_mem=16
enable_uart=1

[peterharperuk]

Sorry - uart_2ndstage=1 has to go in config.txt not cmdline.txt

[framps]

Oops - sorry.

But I still don't get any other debug lines. Neither on the eMMC image nor on the NVMe image.

cmdline.txt

console=serial0,115200 console=tty1 root=PARTUUID=7a1b13df-02 rootfstype=ext4 fsck.repair=yes rootwait

At the end of config.txt I have

[all]
gpu_mem=16
enable_uart=1
uart_2ndstage=1

[framps]

I tried the 64 bit lite image and didn't terminate the dangling boot. All of the sudden I got some additional output (see the attached file).

nvme_boot.txt

The last line is very interesting:

[ 3.072075] Waiting for root device PARTUUID=be6c5e24-02...

sudo blkid 
...
/dev/sde1: LABEL_FATBOOT="boot" LABEL="boot" UUID="A07B-BB23" TYPE="vfat" PARTUUID="be6c5e24-01"
/dev/sde2: LABEL="rootfs" UUID="93282bcd-0b56-4477-aed5-dfb0038f9ca8" TYPE="ext4" PARTUUID="be6c5e24-02"
/dev/sdf1: LABEL_FATBOOT="boot" LABEL="boot" UUID="A07B-BB23" TYPE="vfat" PARTUUID="ac1488a6-01"
/dev/sdf2: LABEL="rootfs" UUID="93282bcd-0b56-4477-aed5-dfb0038f9ca8" TYPE="ext4" PARTUUID="ac1488a6-02"

sudo fdisk -l /dev/sde
Disk /dev/sde: 29.12 GiB, 31268536320 bytes, 61071360 sectors
Disk model:                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xbe6c5e24

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sde1         8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/sde2       532480 61071359 60538880 28.9G 83 Linux

sudo fdisk -l /dev/sdf
Disk /dev/sdf: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Disk model:                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xac1488a6

Device     Boot  Start     End Sectors  Size Id Type
/dev/sdf1         8192  532479  524288  256M  c W95 FAT32 (LBA)
/dev/sdf2       532480 3661823 3129344  1.5G 83 Linux

My eMMC has 32GB and the NVMe has 128GB. This means 1) Boot takes a long time now (approx 30 seconds). Usually it boots after some seconds. 2) At the end of the boot process the eMMC root partition will be looked up as rootfs even the NVMe image boots.

I checked /boot/cmdline,txt and /etc/fstab on the NVMe image and both have the correct UUID of the NVMe partition (ac1488a6-02).

Not sure whether this helps to locate the root cause for the issue. It's also strange I don't get any additional output even I have

gpu_mem=16
enable_uart=1
uart_2ndstage=1

at the end of config.txt.

[framps]

Just to proof my steps I execute in order to get an uart output with 2nd stage output are correct I executed following steps:

1) Restored the 04-04 lite image on eMMC and added the two uart lines in /boot/config.txt. I got 2ndstage output. 2) Restored the 09-06 lite image on eMMC and added the two uart lines in /boot/config.txt. I got again 2ndstage output. 3) Restored the 04-04 lite image on NVMe and added the two uart lines in /boot/config.txt. Next I changed the BOOT_ODER to boot from NVMe. I got 2ndstage output. 4) Restored the 09-06 lite image on NVMe and added the two uart lines in /boot/config.txt. BOOT_ORDER change was not required. I don't get any 2ndstage output :-( . The system just hangs. For me it sounds there is some issue in the startup phase of 2ndstage before any debug statements are executed or any files required for 2nd stage are not found and boot hangs somewhere. I also executed the same steps (4) and updated the eeprom with the boot order again: no 2ndstage output.

I also noticed the system doesn't look for the eMMC rootfs any more after some time what I described in my previous comment.

Please let me know what I should do next in order to help you to locate the root cause for the issue.

[framps]

Maybe the boot messages of the 04-04 image booting from NVMe which include the 2ndstage messages help?

nvme_boot.txt

[peterharperuk]

I ordered this drive and it works fine. It arrived with a slightly different model number BC501A despite me ordering BC711.

Your logs indicate that the bootloader and firmware worked okay. The kernel seems to be stuck waiting for the drive to appear. It's possible the bootloader or firmware are causing issues for the kernel.

Things you could try if you're up for it. 1) A clean install - you will lose any data on the drive however. 2) Boot using the SD (BOOTORDER 1) but set the root parameter in cmdline.txt to point to the NVMe drive (root=PARTUUID=be6c5e24-02). If it still doesn't work then it implies a kernel issue with that drive?

[framps]

I ordered this drive and it works fine. It arrived with a slightly different model number BC501A despite me ordering BC711.

Thank you very much you tried to recreate my issue.

The kernel seems to be stuck waiting for the drive to appear.

It's not a busy loop. The heat sink stays warm as usual. Looks like the kernel waits for an interrupt.

2. set the root parameter in cmdline.txt to point to the NVMe drive

Clean install will be done on NVMe. Should I then change cmdline.txt on the NVMe boot partition or eMMC boot partition? Just guessing eMMC boot partition may be used with BOOTORDER 1and thus cmdline.txt of eMMC boot partition will be used.

I'm still busy right now but I will follow your instructions mid next week.

[framps]

I'm back home now.

That's what I did:

1) Prime the emmc storage with the 09 lite image and set BOOTORDER to 1 to boot from emmc 2) Booted the CM4. Got messages a ssh key is generated. then the system rebooted and got a second time a message a ssh key is generated. Next I was prompted for a user and password. Then I was prompted for the keyboard I changed to a German keyboard. 3) Next I got a login prompt 4) I logged in and verified the system booted from emmc

5) Primed the nvme with the 09 lite image and kept the boot order. 6) Tried to boot - failure 7) Modified cmdline.txt on the emmc storage to use the nvme partuuid of rootfs 8) Tried to reboot - success

So that's what I actually asked for in my previous comment: When I change the BOOTORDER to 1 (SD) the emmc storage will be used and thus the cmdline.txt on this storage has to use the nvme root filesystem.

The interesting part is that the system now uses /boot from the nvme storage

pi@raspberrypi:~ $ mount | grep nvme
/dev/nvme0n1p2 on / type ext4 (rw,noatime)
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

But if I change the BOOTORDER back to 6 now the system doesn't boot any more.

I have no clue what's going on there but it seems to me the discovery of the boot partition on my nvme storage fails. If I use the boot partition of the emmc storage which can be discovered and use the nvme root partition it works.

It's also interesting the ssh key is generated twice. First on the emmc root partition and second on the nvme root partition?

2. If it still doesn't work then it implies a kernel issue with that drive?

It works - but with a mixture of emmc and nvme partitions. That's the way it's possible to use a USB disk with an older Raspberry which is not USB boot capable: Just insert a SD card with /boot and modify cmdline.txt to use the USB rootpartition.

But that's not the way it should work with nvme. @peterharperuk Did you execute your test with a CM4 which has emmc and nvme or with a CM4 which has no emmc but nvme?

[peterharperuk]

I probably used a cm4 lite. If I send you a test bootloader/firmware, are you able test something for me? I don't really know why switching release would make a difference, but we have recently spotted a timing issue ultimately caused by us not disabling nvme after we've finished with it.

[framps]

I probably used a cm4 lite.

Ok. So my issue may not be an issue with my hynix nvme card but the fact I have emmc and nvme storage. Maybe somebody can test this configuration to verify it's not an issue of a cm4 equipped with eMMC and NVMe?

If I send you a test bootloader/firmware, are you able test something for me?

Sure. Just give me a download URL and instructions how to flash this code on my cm4. Please also let me know how I can revert and go back to the current bootloader/firmware - just in case ... Should I save the current code first? If yes, I need instructions how to do this.

[peterharperuk]

pieeprom.original.tipwithfix.zip

The attached has a bootloader and firmware. Copy start4.elf and fixup4.dat to the /boot partition on the nvme drive. Backup the old files. Update the eeprom with the pieeeprom* image.

programming the bootloader...

To program the bootloader you need to use usbboot git@github.com:raspberrypi/usbboot.git. Fetch this e.g. on a rpi4 and build the code by running make. Copy the new bootloader into the usbboot/recovery folder. You have to replace the file pieeprom.original.bin. Sign the bootloader and run rpiboot, like this...

cd usbboot/recovery ./update-pieeprom.sh ../rpiboot -d .

Connect the rpi4 to the USB slave and disable eMMC boot with a jumper (the Waveshare board has a switch for this and uses usb c cable connected to its power port) . The cm4 will try booting via rpiboot, it should connect to the rpi4 and program the eeprom with the new bootloader. The screen goes green if this was successful. Re-enable eMMC boot by removing the jumper and remove the USB cable. It should then boot with the new bootloader after a powercycle.

You should see the following text on boot. I added the "pete" thing so you can confirm that you've updated your bootloader correctly. You can set your boot order back to 6 and set BOOT_UART=0

pete: f80b151a 0x00c03140 0x00000000 0x00001fff MEM GPU: 76 ARM: 948 TOTAL: 1024 Starting start4.elf @ 0xfec00200 partition 0 NVME off

[framps]

That's what I did:

1) Primed the nvme storage with the 09 lite image 2) Saved old start4.elf and copied your start4.elf into /boot 3) Saved fixup4.dat and copied your fixup4.dat into /boot 4) Saved old pieeprom.original.bin and copied pieeprom.original.tipwithfix.bin into /usbboot/recovery as pieeprom.original.bin 5) Executed on my Linux desktop ./update-pieeprom.sh ../rpiboot -d . 6) Rebooted cm4 with bootorder 1. Boot hangs. See attached UART log nvme_bootorder_1.log

7) Changed bootorder to 6 8) Rebooted cm4 with bootorder 6. Boot hangs. See attached UART log nvme_bootorder_6.log

As you can see in the logs your bootloader pete is used. :smiley:

The second log shows NVME off but this doesn't fix the issue. It even doesn't boot with bootmode 1 from eMMC :cry:

0x00c03140 0x00000000 0x00001fff

I noticed this line is different than yours 0x00a03140 0x00000000 0x00001fff.

Anything else I can do for you to locate the root cause of the issue?

[framps]

I just wanted to revert all changes and detected I updated start4.elf and fixup4.dat on the emmc boot partition by error :worried:

Then I copied these files on the nvme boot partition and rebooted cm4 again (I still have bootorder 6).

cm4 booted, generated ssh key once - not twice what happens without the fix - and asked for keyboard layout and uid and pwd of user.

Success :smile: Your patch fixed the issue :+1:

Do I have to wait for an official fix now or can I continue and use this patched cm4 and any future updates will replace the patched code?

[peterharperuk]

Future updates should replace those files.

[framps]

Great.

I just primed both emmc and nvme storage with the 09 image and updated start4.elf and fixup4.dat on both boot partitions in order to make sure there is no regression with emmc boot.

Everything works fine now.

But I noticed during first boot from emmc the ssh key was generated twice. And when I booted from nvme there was no ssh key generated any more. Looks like the ssh generation is executed on all available rootfs during first boot. Not sure whether this works as designed ...

@leidrin You may be interested in this issue :wink:

[framps]

I just used raspi-config and configured

1) 16 GPU size 2) Changed hostname to raspberrypi-buster-nvme 3) Add EN-US language 4) Set timezone to Berlin 5) Resized nvme partition to use whole space

The system reboots and hangs now again

[framps]

I nailed the boot issue down to the GPU memory size reduction. With the previous image I was able to reduce the GPU memory.

Not sure whether this is related to the nvme boot issue or whether it's another issue in the 09 image.

[peterharperuk]

Why are you messing with GPU memory?

[framps]

I run all my Raspberries headless and just need the minimal GPU memory. If it's no longer supported to lower the GPU memory to 16 MB with 09 image and more memory is required with the 09 image this has to be documented and changed in raspi-config accordingly.

[framps]

I primed eMMC and NMVe with the same 09 image and your fix. Note I can change GPU memory to 16MB on eMMC without any boot issue.

[timg236]

If gpu_mem in config.txt is < 32 then startcd4.elf is loaded instead of start4.elf so you will be seeing the old firmware release.

[framps]

Sounds reasonable. Please provide a fixed version of startcd4.elf and I will verify the fix :wink:

[peterharperuk]

If you run rpi-update you'll get the fix to all versions of start*.elf

[framps]

Thank you very much for the update.

That's what I did now to verify the fix:

1) Flashed 0404 image on nvme storage 2) Booted system 3) Executed sudo rpi-update 4) Executed sudo apt-get update; sudo apt-get -upgrade 5) Kept the default GPU memory 6) Rebooted the CM4 No boot. System hangs :disappointed:

I checked the UART log and detected there are still the debug messages pete and NVME off.

RPi: BOOTLOADER release VERSION:3165691d DATE: 2022/09/29 TIME: 15:56:43 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1664463403 0x34a87b72 0x00a03140 0x000747a9
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 8Gb x1 total-size: 8 Gbit 3200
DDR 3200 0 0 8 152

Boot mode: NVME (06) order f2514
VID 0x1c5c MN BC711 NVMe SK hynix 128GB              
NVME on
MBR: 0x00002000,  524288 type: 0x0c
MBR: 0x00082000, 3129344 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 1020 c-count 130554 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 130554
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 1020 c-count 130554 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 130554
Read config.txt bytes     2075 hnd 0x163
Read start4.elf bytes  2249280 hnd 0x3cdb
Read fixup4.dat bytes     5399 hnd 0x169
pete: f80b151a
0x00a03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 102f1e848393c2112206fadffaaf86db04e98326 Aug 26 2022 14:03:16
Starting start4.elf @ 0xfec00200 partition 0
NVME off
+

Not sure whether the following output helps which I created when the rpi-update finished:

root@raspberrypi:/boot# ls -la
total 49851
drwxr-xr-x  3 root root    4096 Jan  1  1970 .
drwxr-xr-x 19 root root    4096 Oct  6 16:49 ..
-rwxr-xr-x  1 root root      41 Oct  6 16:51 .firmware_revision
-rwxr-xr-x  1 root root   18693 Mar 31  2022 COPYING.linux
-rwxr-xr-x  1 root root    1594 Mar 31  2022 LICENCE.broadcom
-rwxr-xr-x  1 root root   28418 Oct  6 16:50 bcm2708-rpi-b-plus.dtb
-rwxr-xr-x  1 root root   27790 Oct  6 16:50 bcm2708-rpi-b-rev1.dtb
-rwxr-xr-x  1 root root   28111 Oct  6 16:50 bcm2708-rpi-b.dtb
-rwxr-xr-x  1 root root   28032 Oct  6 16:50 bcm2708-rpi-cm.dtb
-rwxr-xr-x  1 root root   29267 Oct  6 16:50 bcm2708-rpi-zero-w.dtb
-rwxr-xr-x  1 root root   27856 Oct  6 16:50 bcm2708-rpi-zero.dtb
-rwxr-xr-x  1 root root   29305 Oct  6 16:50 bcm2709-rpi-2-b.dtb
-rwxr-xr-x  1 root root   29304 Oct  6 16:50 bcm2709-rpi-cm2.dtb
-rwxr-xr-x  1 root root   30170 Oct  6 16:50 bcm2710-rpi-2-b.dtb
-rwxr-xr-x  1 root root   32533 Oct  6 16:50 bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x  1 root root   31922 Oct  6 16:50 bcm2710-rpi-3-b.dtb
-rwxr-xr-x  1 root root   30157 Oct  6 16:50 bcm2710-rpi-cm3.dtb
-rwxr-xr-x  1 root root   31230 Oct  6 16:50 bcm2710-rpi-zero-2-w.dtb
-rwxr-xr-x  1 root root   31230 Oct  6 16:50 bcm2710-rpi-zero-2.dtb
-rwxr-xr-x  1 root root   52052 Oct  6 16:50 bcm2711-rpi-4-b.dtb
-rwxr-xr-x  1 root root   52184 Oct  6 16:50 bcm2711-rpi-400.dtb
-rwxr-xr-x  1 root root   52637 Oct  6 16:50 bcm2711-rpi-cm4.dtb
-rwxr-xr-x  1 root root   49951 Oct  6 16:50 bcm2711-rpi-cm4s.dtb
-rwxr-xr-x  1 root root   52460 Oct  6 16:50 bootcode.bin
-rwxr-xr-x  1 root root     103 Jan  1  1980 cmdline.txt
-rwxr-xr-x  1 root root    2075 Apr  4  2022 config.txt
-rwxr-xr-x  1 root root    7262 Oct  6 16:50 fixup.dat
-rwxr-xr-x  1 root root    5395 Oct  6 16:50 fixup4.dat
-rwxr-xr-x  1 root root    3170 Oct  6 16:50 fixup4cd.dat
-rwxr-xr-x  1 root root    8379 Oct  6 16:50 fixup4db.dat
-rwxr-xr-x  1 root root    8379 Oct  6 16:50 fixup4x.dat
-rwxr-xr-x  1 root root    3170 Oct  6 16:50 fixup_cd.dat
-rwxr-xr-x  1 root root   10226 Oct  6 16:50 fixup_db.dat
-rwxr-xr-x  1 root root   10224 Oct  6 16:50 fixup_x.dat
-rwxr-xr-x  1 root root     145 Apr  4  2022 issue.txt
-rwxr-xr-x  1 root root 6269448 Oct  6 16:50 kernel.img
-rwxr-xr-x  1 root root 6630752 Oct  6 16:50 kernel7.img
-rwxr-xr-x  1 root root 7040648 Oct  6 16:50 kernel7l.img
-rwxr-xr-x  1 root root 8193235 Oct  6 16:50 kernel8.img
drwxr-xr-x  2 root root   24064 Oct  6 16:50 overlays
-rwxr-xr-x  1 root root 2974112 Oct  6 16:50 start.elf
-rwxr-xr-x  1 root root 2249856 Oct  6 16:50 start4.elf
-rwxr-xr-x  1 root root  804380 Oct  6 16:50 start4cd.elf
-rwxr-xr-x  1 root root 3746024 Oct  6 16:50 start4db.elf
-rwxr-xr-x  1 root root 2997320 Oct  6 16:50 start4x.elf
-rwxr-xr-x  1 root root  804380 Oct  6 16:50 start_cd.elf
-rwxr-xr-x  1 root root 4817896 Oct  6 16:50 start_db.elf
-rwxr-xr-x  1 root root 3720968 Oct  6 16:50 start_x.elf
root@raspberrypi:/boot# cat .firmware_revision 
9bd03428a49f638b4a2c57015d0f7aea49cf16fa

[peterharperuk]

This doesn't appear to match the version of start4.elf you have in your boot folder? Read start4.elf bytes 2249280 hnd 0x3cdb Firmware: 102f1e848393c2112206fadffaaf86db04e98326 Aug 26 2022 14:03:16

It should show... Read start4.elf bytes 2249856 hnd 0x6c2d Firmware: 6c57d97fa8066929a2fee6e4fac6d64ad62506c9 Oct 5 2022 15:51:11

[peterharperuk]

you'll see the "pete" line until you update your bootloader. We have a beta in apt but we've found an issue with it, so you might want to hold on a bit.

[framps]

This doesn't appear to match the version of start4.elf you have in your boot folder?

Good catch :+1:

I executed steps 1-3 again and checked UART log again. Now the filesize of start4.elf in /boot matches the UART boot log. I have no idea why it didn't match in my first attempt.

RPi: BOOTLOADER release VERSION:3165691d DATE: 2022/09/29 TIME: 15:56:43 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1664463403 0x34a87b72 0x00a03140 0x00074127
PM_RSTS: 0x00000020
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 8Gb x1 total-size: 8 Gbit 3200
DDR 3200 0 0 8 152

Boot mode: NVME (06) order f2514
VID 0x1c5c MN BC711 NVMe SK hynix 128GB              
NVME on
MBR: 0x00002000,  524288 type: 0x0c
MBR: 0x00082000,249537200 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Read config.txt bytes     2075 hnd 0x51b
Read start4.elf bytes  2249856 hnd 0x1a08d
Read fixup4.dat bytes     5395 hnd 0x232a9
pete: f80b151a
0x00a03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 6c57d97fa8066929a2fee6e4fac6d64ad62506c9 Oct  5 2022 15:51:11
Starting start4.elf @ 0xfec00200 partition 0
NVME off
+

Then I executed sudo apt-get update; sudo apt-get upgrade but didn't reboot the system. (I forgot to add this step in my list above and updated the list right now). Now there is an old start4.elf in /boot.

-rwxr-xr-x 1 root root 2249280 Oct  6 18:37 /boot/start4.elf

Looks like the apt upgrade reverted the update of start4.elf because now when I restart the CM4 it hangs.

We have a beta in apt but we've found an issue with it, so you might want to hold on a bit.

Take your time :wink:

[framps]

It's OT now @peterharperuk - but maybe you can help me to get access to my eMMC again. I just executed sudo dd if=/dev/zero of=/dev/sdf bs=2GiB count=10 to my 32GB eMMC (maybe it was count=20). My command window with the dd command hung and I had to force close the window.

I thought I should clean the eMMC storage to make sure there is no code retrieved from there during boot in case of some storage lookup error. Now eMMC doesn't show up if I use mass-storage-gadget nor if I boot from nvme and check with sudo blkid.

Any help is greatly appreciated.

[peterharperuk]

RPI Image has an option to Erase / format a card as Fat32. Try that.

[framps]

Thank you very much for your reply on my OT question.

Please note: It's the internal eMMC, no SD card.

That's what I tried now: I downloaded a fresh usbboot git repo from the raspberry git, executed make and followed the README in rpi-imager-embedded. But the system boots from nvme :cry: .

Given it's OT: Is there any place I can open an issue on this or can ask for help? A RaspberryForum thread or to open an issue in some git repo? I think we shouldn't have OT discussions in this issue.

[framps]

eMMC storage is back again now :smile:

So now I wait for the beta.

[framps]

Is there any outlook for the beta?

[timg236]

The rpi-eeprom APT package was updated today.

[framps]

Great :+1:

Not sure what I have to do now: Should I run rpi-update? Should I run rpi-eeprom-update from the eeprom repo? Or is there anything else I have to do in order to grab the beta?

[peterharperuk]

rpi-update will give you the latest kernel (irrelevant to this problem) and update the "firmware" in the /boot folder (which is relevant to this problem). The /boot "firmware" fix was released a few weeks ago via rpi-update.

The rpi-eeprom APT package contains the beta bootloader. If you "sudo apt update" then "sudo apt upgrade" it updates the bootloader binaries in /lib/firmware/raspberrypi/bootloader. That's fairly irrelevant for a cm4 as the only way to update a cm4 bootloader is using usbboot.

You can do something like this to update the cm4 bootloader... git clone https://github.com/raspberrypi/rpi-eeprom.git cp rpi-eeprom/firmware/beta/pieeprom-2022-10-18.bin usbboot/recovery/pieeprom.original.bin cd usbboot/recovery ./update-pieeprom.sh ../rpiboot -d.

Or wait until we update the bootloader in usbboot.

[framps]

That's what I did right now:

1) Installed 04-04 lite OS 2) Executed rpi-update 3) Cloned rpi-eeprom and executed the steps you described above with rpiboot

Reboot was possible. Next I executed sudo apt-get update; sudo apt-get upgrade to get the latest code.

Reboot was no longer possible :cry:

That's the UART output I get:

Pi: BOOTLOADER release VERSION:23aa699d DATE: 2022/10/18 TIME: 11:57:44 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1666090664 0x34a87b72 0x00a03140 0x0007419c
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 8Gb x1 total-size: 8 Gbit 3200
DDR 3200 0 0 8 152

Boot mode: NVME (06) order f2514
VID 0x1c5c MN BC711 NVMe SK hynix 128GB              
NVME on
MBR: 0x00002000,  524288 type: 0x0c
MBR: 0x00082000,249537200 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' NO NAME    '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' NO NAME    '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 516190
Read config.txt bytes     2075 hnd 0x51b
Read start4.elf bytes  2249280 hnd 0x456c5
Read fixup4.dat bytes     5399 hnd 0x4a182
0x00a03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 102f1e848393c2112206fadffaaf86db04e98326 Aug 26 2022 14:03:16
Starting start4.elf @ 0xfec00200 partition 0
NVME off
+

[peterharperuk]

For some reason you don't have the latest firmware. This line should show something after "Oct 5 2022"

Firmware: 102f1e848393c2112206fadffaaf86db04e98326 Aug 26 2022 14:03:16