raspberrypi / rpi-eeprom

Installation scripts and binaries for the Raspberry Pi 4 and Raspberry Pi 5 bootloader EEPROMs
https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#raspberry-pi-boot-eeprom
Other
1.28k stars 206 forks source link

Rpi 5 NVMe boots only if USB-MSD is there #629

Open doubleuerner opened 2 weeks ago

doubleuerner commented 2 weeks ago

Describe the bug

I am struggling with booting from an NVMe disk (Toshiba KBG40ZNS256G) and after a couple of mistakes and spending fun but also confusing hours i think i discoverd a bug or a symptom caused by doing something wrong or missing something , or it is a bug for real. Who knows? I'll try to explain:

I found out that simply using dd an image to a NVMen0 (4096 sector size) with bs=515(default) gives a weird situation. But the SD card copier tool helped to overcome.

Some info about my setup:

Raspberry Pi 5 Model B Rev 1.0 PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)"

picture:

I've cloned USB-MSD into NVMe disk using the "SD card Copier" tool. I changed the boot order,

BOOT_UART=1 POWER_OFF_ON_HALT=0 BOOT_ORDER=0xf461 NET_INSTALL_AT_POWER_ON=1

During power up SD card is not present, USB thumbdrive neither.

I would expect to boot from NVMe, since i changed the boot order: .

What i get is a pi that doesnt boot and in the diagnotic screen:

picture:

nvme: error 10

But now the suprise:

But powering up with a USB-MSD in one of thew usb ports..., it boots from NVMe, it's so fast you could even not see any 'bios' screen or diagnostic screen.

I am happy my goal has been reached, but only to trick the pi with a USB-MSD first. after start up it is not mounted and i can just remove it.

root@rpi5-usb:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 1 58.6G 0 disk ├─sda1 8:1 1 512M 0 part └─sda2 8:2 1 58.1G 0 part nvme0n1 259:0 0 238.5G 0 disk ├─nvme0n1p1 259:1 0 4G 0 part /boot/firmware └─nvme0n1p2 259:2 0 234.4G 0 part /

It's not the case a config.txt or cmdline.txt is read from USB-MSD, because it requires then to do something with UUID or correct devices (eg sda / sdb)

The last thing i want now is to get rid of the USB-MSD to be injected as a prerequisit to boot the NVMe ... or its a bug may be?

Steps to reproduce the behaviour

NVMe disk partitioned and having OS installed.

power up without SD ; but with USB-MSD : it boot from NVMe disk

power up without SD ; and NO USB-MSD : nvme error 10, boot fails and others (SD, USB ) are try out. Afbeelding van WhatsApp op 2024-11-07 om 14 43 36_ce56ab00 Afbeelding van WhatsApp op 2024-11-07 om 14 24 13_01ec40ac

Device (s)

Raspberry Pi 5

Bootloader configuration.

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# cat pieeprom-backup-20241107-140225.conf [all] BOOT_UART=1 POWER_OFF_ON_HALT=0 BOOT_ORDER=0xf461 NET_INSTALL_AT_POWER_ON=1

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# rpi-eeprom-config [all] BOOT_UART=1 POWER_OFF_ON_HALT=0 BOOT_ORDER=0xf461 NET_INSTALL_AT_POWER_ON=1

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# ls -alh total 20K drwxr-xr-x 2 root root 4.0K Nov 7 14:02 . drwxr-xr-x 3 root root 4.0K Oct 22 15:05 .. -rw------- 1 root root 56 Nov 7 13:59 pieeprom-backup-20241107-135912.conf -rw------- 1 root root 82 Nov 7 13:59 pieeprom-backup-20241107-135940.conf -rw------- 1 root root 82 Nov 7 14:02 pieeprom-backup-20241107-140225.conf root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup#

root@rpi5-usb:~# rpi-eeprom-update BOOTLOADER: up to date CURRENT: Tue 5 Nov 12:38:12 UTC 2024 (1730810292) LATEST: Tue 5 Nov 12:38:12 UTC 2024 (1730810292) RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest) Use raspi-config to change the release.

System

root@rpi5-usb:~# cat /etc/rpi-issue Raspberry Pi reference 2024-10-22 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, ecda2579ebbc125b321984c571e3128f6fd959d4, stage4 root@rpi5-usb:~# vcgencmd bootloader_version 2024/11/05 12:38:12 version 3c4fc886f1f3b34a794d7157e0f26ec5942408ca (release) timestamp 1730810292 update-time 1730984545 capabilities 0x0000007f

root@rpi5-usb:~# vcgencmd version 2024/11/05 12:38:12 Copyright (c) 2012 Broadcom version 3c4fc886 (release) (embedded) root@rpi5-usb:~# uname -a Linux rpi5-usb 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux root@rpi5-usb:~#

Bootloader logs

No response

USB boot

root@rpi5-usb:~# lsusb -vvv

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 3.00 bDeviceClass 9 Hub bDeviceSubClass 0 bDeviceProtocol 3 bMaxPacketSize0 9 idVendor 0x1d6b Linux Foundation idProduct 0x0003 3.0 root hub bcdDevice 6.06 iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd iProduct 2 xHCI Host Controller iSerial 1 xhci-hcd.1 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x001f bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 bMaxBurst 0 Hub Descriptor: bLength 12 bDescriptorType 42 nNbrPorts 1 wHubCharacteristic 0x0009 Per-port power switching Per-port overcurrent protection bPwrOn2PwrGood 50 * 2 milli seconds bHubContrCurrent 0 milli Ampere bHubDecLat 0.0 micro seconds wHubDelay 0 nano seconds DeviceRemovable 0x00 Hub Port Status: Port 1: 0000.02a0 5Gbps power Rx.Detect Binary Object Store Descriptor: bLength 5 bDescriptorType 15 wTotalLength 0x000f bNumDeviceCaps 1 SuperSpeed USB Device Capability: bLength 10 bDescriptorType 16 bDevCapabilityType 3 bmAttributes 0x02 Latency Tolerance Messages (LTM) Supported wSpeedsSupported 0x0008 Device can operate at SuperSpeed (5Gbps) bFunctionalitySupport 1 Lowest fully-functional device speed is Full Speed (12Mbps) bU1DevExitLat 10 micro seconds bU2DevExitLat 2047 micro seconds can't get debug descriptor: Resource temporarily unavailable Device Status: 0x0001 Self Powered

Bus 003 Device 002: ID 346d:5678 Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 0 bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x346d idProduct 0x5678 bcdDevice 2.00 iManufacturer 1 iProduct 2 iSerial 3 8649321074347787162 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x0020 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0x80 (Bus Powered) MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 2 bInterfaceClass 8 Mass Storage bInterfaceSubClass 6 SCSI bInterfaceProtocol 80 Bulk-Only iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x01 EP 1 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Device Qualifier (for other device speed): bLength 10 bDescriptorType 6 bcdUSB 2.00 bDeviceClass 0 bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 bNumConfigurations 1 can't get debug descriptor: Resource temporarily unavailable Device Status: 0x0000 (Bus Powered)

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 9 Hub bDeviceSubClass 0 bDeviceProtocol 1 Single TT bMaxPacketSize0 64 idVendor 0x1d6b Linux Foundation idProduct 0x0002 2.0 root hub bcdDevice 6.06 iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd iProduct 2 xHCI Host Controller iSerial 1 xhci-hcd.1 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x0019 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 Hub Descriptor: bLength 9 bDescriptorType 41 nNbrPorts 2 wHubCharacteristic 0x0009 Per-port power switching Per-port overcurrent protection TT think time 8 FS bits bPwrOn2PwrGood 10 * 2 milli seconds bHubContrCurrent 0 milli Ampere DeviceRemovable 0x00 PortPwrCtrlMask 0xff Hub Port Status: Port 1: 0000.0100 power Port 2: 0000.0503 highspeed power enable connect can't get device qualifier: Resource temporarily unavailable can't get debug descriptor: Resource temporarily unavailable Device Status: 0x0001 Self Powered

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 3.00 bDeviceClass 9 Hub bDeviceSubClass 0 bDeviceProtocol 3 bMaxPacketSize0 9 idVendor 0x1d6b Linux Foundation idProduct 0x0003 3.0 root hub bcdDevice 6.06 iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd iProduct 2 xHCI Host Controller iSerial 1 xhci-hcd.0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x001f bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 bMaxBurst 0 Hub Descriptor: bLength 12 bDescriptorType 42 nNbrPorts 1 wHubCharacteristic 0x0009 Per-port power switching Per-port overcurrent protection bPwrOn2PwrGood 50 * 2 milli seconds bHubContrCurrent 0 milli Ampere bHubDecLat 0.0 micro seconds wHubDelay 0 nano seconds DeviceRemovable 0x00 Hub Port Status: Port 1: 0000.02a0 5Gbps power Rx.Detect Binary Object Store Descriptor: bLength 5 bDescriptorType 15 wTotalLength 0x000f bNumDeviceCaps 1 SuperSpeed USB Device Capability: bLength 10 bDescriptorType 16 bDevCapabilityType 3 bmAttributes 0x02 Latency Tolerance Messages (LTM) Supported wSpeedsSupported 0x0008 Device can operate at SuperSpeed (5Gbps) bFunctionalitySupport 1 Lowest fully-functional device speed is Full Speed (12Mbps) bU1DevExitLat 10 micro seconds bU2DevExitLat 2047 micro seconds can't get debug descriptor: Resource temporarily unavailable Device Status: 0x0001 Self Powered

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 9 Hub bDeviceSubClass 0 bDeviceProtocol 1 Single TT bMaxPacketSize0 64 idVendor 0x1d6b Linux Foundation idProduct 0x0002 2.0 root hub bcdDevice 6.06 iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd iProduct 2 xHCI Host Controller iSerial 1 xhci-hcd.0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x0019 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0004 1x 4 bytes bInterval 12 Hub Descriptor: bLength 9 bDescriptorType 41 nNbrPorts 2 wHubCharacteristic 0x0009 Per-port power switching Per-port overcurrent protection TT think time 8 FS bits bPwrOn2PwrGood 10 * 2 milli seconds bHubContrCurrent 0 milli Ampere DeviceRemovable 0x00 PortPwrCtrlMask 0xff Hub Port Status: Port 1: 0000.0100 power Port 2: 0000.0100 power can't get device qualifier: Resource temporarily unavailable can't get debug descriptor: Resource temporarily unavailable Device Status: 0x0001 Self Powered root@rpi5-usb:~#

NVMe boot

Node Generic SN Model Namespace Usage Format FW Rev


/dev/nvme0n1 /dev/ng0n1 799PE0N0PQEN KBG40ZNS256G NVMe TOSHIBA 256GB 1 256.06 GB / 256.06 GB 4 KiB + 0 B 10410106

root@rpi5-usb:~# nvme id-ctrl -H /dev/nvme0 NVME Identify Controller: vid : 0x1e0f ssvid : 0x1e0f sn : 799PE0N0PQEN mn : KBG40ZNS256G NVMe TOSHIBA 256GB fr : 10410106 rab : 3 ieee : 8ce38e cmic : 0 [3:3] : 0 ANA not supported [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port

mdts : 9 cntlid : 0 ver : 0x10300 rtd3r : 0x7a120 rtd3e : 0x2bf20 oaes : 0x200 [31:31] : 0 Discovery Log Change Notice Not Supported [27:27] : 0 Zone Descriptor Changed Notices Not Supported [15:15] : 0 Normal NSS Shutdown Event Not Supported [14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported [13:13] : 0 LBA Status Information Notices Not Supported [12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported [11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported [9:9] : 0x1 Firmware Activation Notices Supported [8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0x2 [19:19] : 0 Flexible Data Placement Not Supported [15:15] : 0 Extended LBA Formats Not Supported [14:14] : 0 Delete NVM Set Not Supported [13:13] : 0 Delete Endurance Group Not Supported [12:12] : 0 Variable Capacity Management Not Supported [11:11] : 0 Fixed Capacity Management Not Supported [10:10] : 0 Multi Domain Subsystem Not Supported [9:9] : 0 UUID List Not Supported [8:8] : 0 SQ Associations Not Supported [7:7] : 0 Namespace Granularity Not Supported [6:6] : 0 Traffic Based Keep Alive Not Supported [5:5] : 0 Predictable Latency Mode Not Supported [4:4] : 0 Endurance Groups Not Supported [3:3] : 0 Read Recovery Levels Not Supported [2:2] : 0 NVM Sets Not Supported [1:1] : 0x1 Non-Operational Power State Permissive Supported [0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0 cntrltype : 0 [7:2] : 0 Reserved [1:0] : 0 Controller type not reported fguid : 00000000-0000-0000-0000-000000000000 crdt1 : 0 crdt2 : 0 crdt3 : 0 nvmsr : 0 [1:1] : 0 NVM subsystem Not part of an Enclosure [0:0] : 0 NVM subsystem Not part of an Storage Device

vwci : 0 [7:7] : 0 VPD Write Cycles Remaining field is Not valid. [6:0] : 0 VPD Write Cycles Remaining

mec : 0 [1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port [0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port

oacs : 0x17 [10:10] : 0 Lockdown Command and Feature Not Supported [9:9] : 0 Get LBA Status Capability Not Supported [8:8] : 0 Doorbell Buffer Config Not Supported [7:7] : 0 Virtualization Management Not Supported [6:6] : 0 NVMe-MI Send and Receive Not Supported [5:5] : 0 Directives Not Supported [4:4] : 0x1 Device Self-test Supported [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Security Send and Receive Supported

acl : 3 aerl : 7 frmw : 0x14 [5:5] : 0 Multiple FW or Boot Update Detection Not Supported [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x2 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0xe [6:6] : 0 Telemetry Log Data Area 4 Not Supported [5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported [4:4] : 0 Persistent Event log Not Supported [3:3] : 0x1 Telemetry host/controller initiated log page Supported [2:2] : 0x1 Extended data for Get Log Page Supported [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 255 [7:0] : 255 (0's based) Error Log Page Entries (ELPE)

npss : 4 [7:0] : 4 (0's based) Number of Power States Support (NPSS)

avscc : 0 [0:0] : 0 Admin Vendor Specific Commands uses Vendor Specific Format

apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported

wctemp : 356 [15:0] : 83°C (356 Kelvin) Warning Composite Temperature Threshold (WCTEMP)

cctemp : 360 [15:0] : 87°C (360 Kelvin) Critical Composite Temperature Threshold (CCTEMP)

mtfa : 0 hmpre : 15616 hmmin : 5888 tnvmcap : 256,060,514,304 [127:0] : 256,060,514,304 Total NVM Capacity (TNVMCAP)

unvmcap : 0 [127:0] : 0 Unallocated NVM Capacity (UNVMCAP)

rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units

edstt : 11 dsto : 1 fwug : 1 kas : 0 hctma : 0x1 [0:0] : 0x1 Host Controlled Thermal Management Supported

mntmt : 0 [15:0] : -273°C (0 Kelvin) Minimum Thermal Management Temperature (MNTMT)

mxtmt : 356 [15:0] : 83°C (356 Kelvin) Maximum Thermal Management Temperature (MXTMT)

sanicap : 0x2 [31:30] : 0 Additional media modification after sanitize operation completes successfully is not defined [29:29] : 0 No-Deallocate After Sanitize bit in Sanitize command Supported [2:2] : 0 Overwrite Sanitize Operation Not Supported [1:1] : 0x1 Block Erase Sanitize Operation Supported [0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 0 hmmaxd : 0 nsetidmax : 0 endgidmax : 0 anatt : 0 anacap : 0 [7:7] : 0 Non-zero group ID Not Supported [6:6] : 0 Group ID does change [4:4] : 0 ANA Change state Not Supported [3:3] : 0 ANA Persistent Loss state Not Supported [2:2] : 0 ANA Inaccessible state Not Supported [1:1] : 0 ANA Non-optimized state Not Supported [0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0 nanagrpid : 0 pels : 0 domainid : 0 megcap : 0 sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 0 nn : 1 oncs : 0x5f [8:8] : 0 Copy Not Supported [7:7] : 0 Verify Not Supported [6:6] : 0x1 Timestamp Supported [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported

fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported

fna : 0 [3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s)

vwc : 0x1 [2:1] : 0 Support for the NSID field set to FFFFFFFFh is not indicated [0:0] : 0x1 Volatile Write Cache Present

awun : 65535 awupf : 0 icsvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format

nwpc : 0 [2:2] : 0 Permanent Write Protect Not Supported [1:1] : 0 Write Protect Until Power Supply Not Supported [0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 0 ocfs : 0 [1:1] : 0 Controller Copy Format 1h Not Supported [0:0] : 0 Controller Copy Format 0h Not Supported

sgls : 0 [15:8] : 0 SGL Descriptor Threshold [1:0] : 0 Scatter-Gather Lists Not Supported

mnan : 0 maxdna : 0 maxcna : 0 subnqn : nqn.2018-06.com.toshiba-memory:KBG40ZNS256G NVMe TOSHIBA 256GB:799PE0N0PQEN ioccsz : 0 iorcsz : 0 icdoff : 0 fcatt : 0 [0:0] : 0 Dynamic Controller Model

msdbd : 0 ofcs : 0 [0:0] : 0 Disconnect command Not Supported

ps 0 : mp:3.60W operational enlat:1 exlat:1 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- active_power_workload:- ps 1 : mp:2.60W operational enlat:1 exlat:1 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- active_power_workload:- ps 2 : mp:2.20W operational enlat:1 exlat:1 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- active_power_workload:- ps 3 : mp:0.0500W non-operational enlat:800 exlat:1200 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- active_power_workload:- ps 4 : mp:0.0050W non-operational enlat:3000 exlat:32000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:- active_power_workload:- root@rpi5-usb:~#

root@rpi5-usb:~# nvme list-ns /dev/nvme0

root@rpi5-usb:~# nvme id-ns -H /dev/nvme0 --namespace-id=1 NVME Identify Namespace 1: nsze : 0x3b9e656 ncap : 0x3b9e656 nuse : 0x3b9e656 nsfeat : 0 [4:4] : 0 NPWG, NPWA, NPDG, NPDA, and NOWS are Not Supported [3:3] : 0 NGUID and EUI64 fields if non-zero, Reused [2:2] : 0 Deallocated or Unwritten Logical Block error Not Supported [1:1] : 0 Namespace uses AWUN, AWUPF, and ACWU [0:0] : 0 Thin Provisioning Not Supported

nlbaf : 1 flbas : 0x1 [6:5] : 0 Most significant 2 bits of Current LBA Format Selected [4:4] : 0 Metadata Transferred in Separate Contiguous Buffer [3:0] : 0x1 Least significant 4 bits of Current LBA Format Selected

mc : 0 [1:1] : 0 Metadata Pointer Not Supported [0:0] : 0 Metadata as Part of Extended Data LBA Not Supported

dpc : 0 [4:4] : 0 Protection Information Transferred as Last 8 Bytes of Metadata Not Supported [3:3] : 0 Protection Information Transferred as First 8 Bytes of Metadata Not Supported [2:2] : 0 Protection Information Type 3 Not Supported [1:1] : 0 Protection Information Type 2 Not Supported [0:0] : 0 Protection Information Type 1 Not Supported

dps : 0 [3:3] : 0 Protection Information is Transferred as Last 8 Bytes of Metadata [2:0] : 0 Protection Information Disabled

nmic : 0 [0:0] : 0 Namespace Multipath Not Capable

rescap : 0 [7:7] : 0 Ignore Existing Key - Used as defined in revision 1.2.1 or earlier [6:6] : 0 Exclusive Access - All Registrants Not Supported [5:5] : 0 Write Exclusive - All Registrants Not Supported [4:4] : 0 Exclusive Access - Registrants Only Not Supported [3:3] : 0 Write Exclusive - Registrants Only Not Supported [2:2] : 0 Exclusive Access Not Supported [1:1] : 0 Write Exclusive Not Supported [0:0] : 0 Persist Through Power Loss Not Supported

fpi : 0 [7:7] : 0 Format Progress Indicator Not Supported

dlfeat : 1 [4:4] : 0 Guard Field of Deallocated Logical Blocks is set to 0xFFFF [3:3] : 0 Deallocate Bit in the Write Zeroes Command is Not Supported [2:0] : 0x1 Bytes Read From a Deallocated Logical Block and its Metadata are 0x00

nawun : 0 nawupf : 0 nacwu : 0 nabsn : 0 nabo : 0 nabspf : 0 noiob : 0 nvmcap : 256,060,514,304 mssrl : 0 mcl : 0 msrc : 0 nulbaf : 0 anagrpid: 0 nsattr : 0 nvmsetid: 0 endgid : 0 nguid : 01000000000000008ce38e040011a073 eui64 : 8ce38e040011a073 LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better (in use) root@rpi5-usb:~#

Network (TFTP boot)

No response

peterharperuk commented 2 weeks ago

My guess is that there's something weird with your boot partition on the NVME drive. Why is it 4G in size? Are the cmdline.txt files the same on the USB disk and the NVME disk? I boot my Pi 5 from NVME all the time, so you've probably done something wrong.

doubleuerner commented 2 weeks ago

well, if i just download the img.xz and unpack it and use dd to write it to NVMe disk, then i get this 4G boot partition. I discovered that dd used 512 and the nvme 4096 blocksize. So then i looked with fdisk into the img-file and see a partition of 512M, and that makes sense because the bs = x8, so 4GB. so i did all by hand using parted/fdisk and mounted and copied all files. Then still i doesnt work, and used the rpi-imager: there the 4GB boot partition was there , no succes. Then used sd card copier, and eventually no succes, until i also used the SUB-MSB stick. And yes the content on NVMe is exact the same as USB. so also cmdline.txt.

peterharperuk commented 2 weeks ago

We don't support 4096 byte block size in the bootloader. That's the problem.

doubleuerner commented 2 weeks ago

Aha, okay then I go change the sector size to 512 and try to dd the img to nvmen1. Thanks for the hint! I'll let you know the result!

https://man.archlinux.org/man/nvme-format.1.en

doubleuerner commented 2 weeks ago

It works!

Thanks , i learned a lot having tried 'everything' and taking all kinds of directions, but finally it works and i understand why.

peterharperuk commented 2 weeks ago

One day we might fix this. It would be interesting to know if performance is noticeably better when the linux partition uses a 4096 byte block size.

peterharperuk commented 2 weeks ago

Actually, reopen. We should probably throw a more obvious error to stop people wasting their time with this in future?

doubleuerner commented 2 weeks ago

Well, I think this struggle appearing to be sector size other than 512 is very common for NVMe's If I just Google on this you see discussion about turning to a different block size in general. The NVMe card i have was from an old windows machine. I think , me as an hobbyist as many others re-use stuff and try to make it work, i don't know if you buy a new NVMe the default is 512 of something else. I thing even the rpi-imager uses dd as underlying method and i know this turns into weird partition table and filesystem not being mount-able.

doubleuerner commented 2 weeks ago

The M.2 HAT+ has documentation explaining it all, but not this exception of 4096 sector size which is not able to boot.

peterharperuk commented 2 weeks ago

Yeah, even I missed "nvme: error 10" in your screen grab which means NMVE_ERROR_UNSUPPORTED_BLOCK_SIZE

timg236 commented 2 weeks ago

The feature is on the todo list (but on the back burner) - to make this useful it will also require updates to RPi Imager / RPi OS for provisioning images in this format.

https://github.com/raspberrypi/firmware/issues/1840

peterharperuk commented 2 weeks ago

I'm just suggesting making the error more obvious for now.

doubleuerner commented 2 weeks ago

Yeah, even I missed "nvme: error 10" in your screen grab which means NMVE_ERROR_UNSUPPORTED_BLOCK_SIZE

google on nvme: error 10 yesterday didn't show up something, also i tried to see if the source code is somewhere. But is this something that can be found by me for instance, or is it something in the code nobody except the developers can find?