Closed ctschach closed 2 years ago
Sounds quite similar to https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/188 ?
I've tried to reproduce this but it seems to work for me. I've tried a LITE and 8GB EMMC CM4. I'm using the waveshare board mentioned in https://github.com/raspberrypi/Raspberry-Pi-OS-64bit/issues/188 that has the resistor fix, but I don't know why that would be relevant. I'm using a different make of NVME disk.
@ctschach if I sent you a version of the bootloader with some extra logging - would you be prepared to update the eeprom on your CM4 using USBBOOT and repro the problem?
@ctschach if I sent you a version of the bootloader with some extra logging - would you be prepared to update the eeprom on your CM4 using USBBOOT and repro the problem?
Absolutely....
Ok, first attempt... pieeprom.original1.zip
cd usbboot\recovery cp pieeprom.original1.bin pieeprom.original.bin ./update-pieeprom.sh ../rpiboot.exe -d .
switch [RPI]boot switch on and connect your PC to the USB type-C interface to program the eeprom with this image. Switch off the boot switch and power cycle and repro the problem. If you can attach any UART logging that would be helpful
So here is what happened: I flashed the new EEPROM and booted the RPi from the NVMe card. This ended up in a kernel panic (see below). Since then the NVMe became unusable. If I boot from an SD card, I see the NVMe disk, but an fsck giving me error. Putting it into an external enclosure and attaching it to another Raspi did not work either.
The interesting thing: It worked directly before changing the EEPROM. I don't think that this could have destroyed the NVMe disk - maybe just some unfortunate coincidences.
I've now ordered two new (and different) disks which should arrive early next week. I'll test them and report back.
´´´
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.78-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1484 SMP Thu Nov 11 16:36:06 GMT 2021
[ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[ 0.000000] random: fast init done
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff]
[ 0.000000] Normal empty
[ 0.000000] HighMem [mem 0x0000000030000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003e5fffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] percpu: Embedded 20 pages/cpu s50828 r8192 d22900 u81920
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 2072384
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:62:3B:5C vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 console=ttyAMA0,115200 console=tty1 root=PARTUUID=0507689b-02 rootfstype=ext4 fsck.repair=yes rootwait
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x0000000012400000-0x0000000016400000] (64MB)
[ 0.000000] Memory: 7809312K/8296448K available (10240K kernel code, 1366K rwdata, 3192K rodata, 2048K init, 883K bss, 159456K reserved, 327680K cma-reserved, 7510016K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] ftrace: allocating 34341 entries in 101 pages
[ 0.000000] ftrace: allocated 101 pages with 4 groups
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 25)
[ 0.000000] random: get_random_bytes called from start_kernel+0x3c8/0x59c with crng_init=1
[ 0.000008] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[ 0.000034] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[ 0.000094] bcm2835: system timer (irq = 26)
[ 0.000746] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[ 0.000767] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[ 0.000789] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[ 0.000807] Switching to timer-based delay loop, resolution 18ns
[ 0.001070] Console: colour dummy device 80x30
[ 0.001834] printk: console [tty1] enabled
[ 0.001903] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=540000)
[ 0.001958] pid_max: default: 32768 minimum: 301
[ 0.002135] LSM: Security Framework initializing
[ 0.002332] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.002378] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.003855] cgroup: Disabling memory control group subsystem
[ 0.004114] CPU: Testing write buffer coherency: ok
[ 0.004589] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.005814] Setting up static identity map for 0x200000 - 0x20003c
[ 0.006028] rcu: Hierarchical SRCU implementation.
[ 0.006972] smp: Bringing up secondary CPUs ...
[ 0.008200] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.009560] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[ 0.010956] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[ 0.011116] smp: Brought up 1 node, 4 CPUs
[ 0.011162] SMP: Total of 4 processors activated (432.00 BogoMIPS).
[ 0.011193] CPU: All CPU(s) started in HYP mode.
[ 0.011220] CPU: Virtualization extensions available.
[ 0.012075] devtmpfs: initialized
[ 0.026254] VFP support v0.3: implementor 41 architecture 3 part 40 variant 8 rev 0
[ 0.026490] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.026543] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.034624] pinctrl core: initialized pinctrl subsystem
[ 0.035739] NET: Registered protocol family 16
[ 0.039779] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[ 0.040600] audit: initializing netlink subsys (disabled)
[ 0.040888] audit: type=2000 audit(0.040:1): state=initialized audit_enabled=0 res=1
[ 0.041474] thermal_sys: Registered thermal governor 'step_wise'
[ 0.042148] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 0.042203] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 0.042622] Serial: AMBA PL011 UART driver
[ 0.085620] bcm2835-mbox fe00b880.mailbox: mailbox enabled
[ 0.100915] raspberrypi-firmware soc:firmware: Attached to firmware from 2021-11-08T18:47:46, variant start_cd
[ 0.110928] raspberrypi-firmware soc:firmware: Firmware hash is 4f73dcaefcfd5b20317e44a81d10e9d74fd3dffe
[ 0.151960] Kprobes globally optimized
[ 0.157392] bcm2835-dma fe007000.dma: DMA legacy API manager, dmachans=0x1
[ 0.161791] vgaarb: loaded
[ 0.162250] SCSI subsystem initialized
[ 0.162494] usbcore: registered new interface driver usbfs
[ 0.162570] usbcore: registered new interface driver hub
[ 0.162671] usbcore: registered new device driver usb
[ 0.163074] usb_phy_generic phy: supply vcc not found, using dummy regulator
[ 0.164994] clocksource: Switched to clocksource arch_sys_counter
[ 1.124459] VFS: Disk quotas dquot_6.6.0
[ 1.124580] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 1.124790] FS-Cache: Loaded
[ 1.125044] CacheFiles: Loaded
[ 1.135421] NET: Registered protocol family 2
[ 1.135685] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 1.136985] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[ 1.137146] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[ 1.137234] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 1.137324] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.137505] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 1.137563] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 1.138249] NET: Registered protocol family 1
[ 1.139016] RPC: Registered named UNIX socket transport module.
[ 1.139049] RPC: Registered udp transport module.
[ 1.139077] RPC: Registered tcp transport module.
[ 1.139106] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.139144] PCI: CLS 0 bytes, default 64
[ 1.142401] Initialise system trusted keyrings
[ 1.142678] workingset: timestamp_bits=14 max_order=21 bucket_order=7
[ 1.151221] zbud: loaded
[ 1.153060] FS-Cache: Netfs 'nfs' registered for caching
[ 1.153862] NFS: Registering the id_resolver key type
[ 1.153925] Key type id_resolver registered
[ 1.153954] Key type id_legacy registered
[ 1.154095] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1.154128] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[ 1.155205] Key type asymmetric registered
[ 1.155237] Asymmetric key parser 'x509' registered
[ 1.155445] bounce: pool size: 64 pages
[ 1.155502] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[ 1.155743] io scheduler mq-deadline registered
[ 1.155775] io scheduler kyber registered
[ 1.160392] gpio-507 (ant1): hogged as output/high
[ 1.161824] gpio-511 (ant2): hogged as output/low
[ 1.163359] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.163408] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.163509] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 1.163618] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[ 1.217101] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[ 1.217523] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.217560] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.217596] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 1.217710] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.217970] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.221453] PCI: bus0: Fast back to back transfers disabled
[ 1.221492] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.221827] pci 0000:01:00.0: [1987:5013] type 00 class 0x010802
[ 1.221919] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 1.222335] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 1.225688] PCI: bus1: Fast back to back transfers disabled
[ 1.225726] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 1.225783] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[ 1.225822] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600003fff 64bit]
[ 1.225887] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 1.225926] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6000fffff]
[ 1.226355] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
[ 1.226604] pcieport 0000:00:00.0: PME: Signaling with IRQ 81
[ 1.234059] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[ 1.237426] iproc-rng200 fe104000.rng: hwrng registered
[ 1.237811] vc-mem: phys_addr:0x00000000 mem_base=0x3f000000 mem_size:0x3f600000(1014 MiB)
[ 1.239319] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
[ 1.251792] brd: module loaded
[ 1.264194] loop: module loaded
[ 1.266089] Loading iSCSI transport class v2.0-870.
[ 1.268180] nvme nvme0: pci function 0000:01:00.0
[ 1.268438] nvme 0000:01:00.0: enabling device (0140 -> 0142)
[ 1.270796] libphy: Fixed MDIO Bus: probed
[ 1.272697] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
[ 1.273494] nvme nvme0: missing or invalid SUBNQN field.
[ 1.273713] nvme nvme0: failed to set APST feature (8578)
[ 1.273818] nvme nvme0: could not set timestamp (8578)
[ 1.285029] libphy: bcmgenet MII bus: probed
[ 1.322199] nvme nvme0: allocated 128 MiB host memory buffer.
[ 1.322337] nvme nvme0: failed to set host mem (err 8578, flags 0x1).
[ 1.385170] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[ 1.386533] usbcore: registered new interface driver r8152
[ 1.386626] usbcore: registered new interface driver lan78xx
[ 1.386704] usbcore: registered new interface driver smsc95xx
[ 1.410145] nvme nvme0: 4/0/0 default/read/poll queues
[ 1.412806] nvme0n1: p1 p2
[ 1.412837] xhci-hcd fe9c0000.xhci: xHCI Host Controller
[ 1.412889] xhci-hcd fe9c0000.xhci: new USB bus registered, assigned bus number 1
[ 1.414140] xhci-hcd fe9c0000.xhci: hcc params 0x0220fe65 hci version 0x110 quirks 0x0000000000010010
[ 1.414264] xhci-hcd fe9c0000.xhci: irq 78, io mem 0xfe9c0000
[ 1.414894] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 1.414935] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.415004] usb usb1: Product: xHCI Host Controller
[ 1.415035] usb usb1: Manufacturer: Linux 5.10.78-v7l+ xhci-hcd
[ 1.415066] usb usb1: SerialNumber: fe9c0000.xhci
[ 1.415784] hub 1-0:1.0: USB hub found
[ 1.415869] hub 1-0:1.0: 1 port detected
[ 1.416417] xhci-hcd fe9c0000.xhci: xHCI Host Controller
[ 1.416461] xhci-hcd fe9c0000.xhci: new USB bus registered, assigned bus number 2
[ 1.416507] xhci-hcd fe9c0000.xhci: Host supports USB 3.0 SuperSpeed
[ 1.416681] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[ 1.416995] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[ 1.417034] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.417070] usb usb2: Product: xHCI Host Controller
[ 1.417099] usb usb2: Manufacturer: Linux 5.10.78-v7l+ xhci-hcd
[ 1.417129] usb usb2: SerialNumber: fe9c0000.xhci
[ 1.417780] hub 2-0:1.0: USB hub found
[ 1.417866] hub 2-0:1.0: config failed, hub doesn't have any ports! (err -19)
[ 1.418731] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 1.419624] usbcore: registered new interface driver uas
[ 1.419753] usbcore: registered new interface driver usb-storage
[ 1.419964] mousedev: PS/2 mouse device common for all mice
[ 1.421805] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 1.425428] sdhci: Secure Digital Host Controller Interface driver
[ 1.425462] sdhci: Copyright(c) Pierre Ossman
[ 1.426191] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.429203] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.429584] hid: raw HID events driver (C) Jiri Kosina
[ 1.429809] usbcore: registered new interface driver usbhid
[ 1.429839] usbhid: USB HID core driver
[ 1.435946] Initializing XFRM netlink socket
[ 1.436010] NET: Registered protocol family 17
[ 1.436138] Key type dns_resolver registered
[ 1.436496] Registering SWP/SWPB emulation handler
[ 1.436682] registered taskstats version 1
[ 1.436723] Loading compiled-in X.509 certificates
[ 1.437599] Key type ._fscrypt registered
[ 1.437630] Key type .fscrypt registered
[ 1.437659] Key type fscrypt-provisioning registered
[ 1.449373] uart-pl011 fe201000.serial: there is not valid maps for state default
[ 1.449653] uart-pl011 fe201000.serial: cts_event_workaround enabled
[ 1.449747] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 38, base_baud = 0) is a PL011 rev2
[ 2.780257] printk: console [ttyAMA0] enabled
[ 2.791688] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[ 2.803529] of_cfs_init
[ 2.806171] of_cfs_init: OK
[ 2.843358] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA
[ 2.856233] EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
[ 2.863577] EXT4-fs (nvme0n1p2): write access will be enabled during recovery
[ 3.045038] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[ 3.223031] blk_update_request: critical medium error, dev nvme0n1, sector 532480 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[ 3.225448] usb 1-1: New USB device found, idVendor=1a40, idProduct=0101, bcdDevice= 1.11
[ 3.234972] Buffer I/O error on dev nvme0n1p2, logical block 0, lost async page write
[ 3.243184] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 3.251018] Buffer I/O error on dev nvme0n1p2, logical block 1, lost async page write
[ 3.258165] usb 1-1: Product: USB 2.0 Hub
[ 3.265994] Buffer I/O error on dev nvme0n1p2, logical block 2, lost async page write
[ 3.266089] blk_update_request: critical medium error, dev nvme0n1, sector 533320 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.289776] Buffer I/O error on dev nvme0n1p2, logical block 105, lost async page write
[ 3.297856] blk_update_request: critical medium error, dev nvme0n1, sector 533416 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.309789] Buffer I/O error on dev nvme0n1p2, logical block 117, lost async page write
[ 3.317824] blk_update_request: critical medium error, dev nvme0n1, sector 533440 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.329757] Buffer I/O error on dev nvme0n1p2, logical block 120, lost async page write
[ 3.337791] blk_update_request: critical medium error, dev nvme0n1, sector 533856 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 3.349723] Buffer I/O error on dev nvme0n1p2, logical block 172, lost async page write
[ 3.357752] Buffer I/O error on dev nvme0n1p2, logical block 173, lost async page write
[ 3.365786] blk_update_request: critical medium error, dev nvme0n1, sector 533920 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[ 3.377717] Buffer I/O error on dev nvme0n1p2, logical block 180, lost async page write
[ 3.385746] Buffer I/O error on dev nvme0n1p2, logical block 181, lost async page write
[ 3.393779] blk_update_request: critical medium error, dev nvme0n1, sector 534080 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.405724] blk_update_request: critical medium error, dev nvme0n1, sector 534456 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.417670] blk_update_request: critical medium error, dev nvme0n1, sector 537704 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.429615] blk_update_request: critical medium error, dev nvme0n1, sector 544992 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3.473628] JBD2: recovery failed
[ 3.477007] EXT4-fs (nvme0n1p2): error loading journal
[ 3.487235] VFS: Cannot open root device "PARTUUID=0507689b-02" or unknown-block(259,2): error -5
[ 3.496159] Please append a correct "root=" boot option; here are the available partitions:
[ 3.504555] 0100 4096 ram0
[ 3.504568] (driver?)
[ 3.510736] 0101 4096 ram1
[ 3.510749] (driver?)
[ 3.516897] 0102 4096 ram2
[ 3.516910] (driver?)
[ 3.523040] 0103 4096 ram3
[ 3.523054] (driver?)
[ 3.529245] 0104 4096 ram4
[ 3.529258] (driver?)
[ 3.535405] 0105 4096 ram5
[ 3.535425] (driver?)
[ 3.541554] 0106 4096 ram6
[ 3.541567] (driver?)
[ 3.547710] 0107 4096 ram7
[ 3.547723] (driver?)
[ 3.553851] 0108 4096 ram8
[ 3.553864] (driver?)
[ 3.560010] 0109 4096 ram9
[ 3.560023] (driver?)
[ 3.566167] 010a 4096 ram10
[ 3.566180] (driver?)
[ 3.572396] 010b 4096 ram11
[ 3.572409] (driver?)
[ 3.578641] 010c 4096 ram12
[ 3.578654] (driver?)
[ 3.584870] 010d 4096 ram13
[ 3.584883] (driver?)
[ 3.591112] 010e 4096 ram14
[ 3.591125] (driver?)
[ 3.597358] 010f 4096 ram15
[ 3.597371] (driver?)
[ 3.603608] 103:00000 500107608 nvme0n1
[ 3.603622] (driver?)
[ 3.610025] 103:00001 262144 nvme0n1p1 0507689b-01
[ 3.610038]
[ 3.616879] 103:00002 499841368 nvme0n1p2 0507689b-02
[ 3.616892]
[ 3.623720] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(259,2)
[ 3.632179] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.10.78-v7l+ #1484
[ 3.638891] Hardware name: BCM2711
[ 3.642301] Backtrace:
[ 3.644781] [
Oh dear... I have one of your disks on order so I'll be testing that. I really can't imagine why that bootloader would cause a problem. It's the same code as the stable build with debug logging enabled in a couple of modules.
I've got the SSD you mentioned and can reproduce the problem you were seeing.
I can repro the issue on a RPI IO board so it doesn't seem to be an issue with the Waveshare board.
I can repro the issue on a RPI IO board so it doesn't seem to be an issue with the Waveshare board.
No sure if I'm allowed to say: "great that you are facing the same problem" :-)
But definitely good that it's wasn't my fault. My other two disks are still in transit. I'll definitely test them both and will report back.
The hardware is indicating a pcie error which seems to "appear" after initialisation. To recover we have to re-initialise the hardware. I have a change that fixes the issue for me - can you give it a try to confirm? pieeprom.original2.zip
Are you sure this is the correct EEPROM? I
I do see an "Unknown boot mode 0x6. Trying SD." in the output on the console. I though the I need boot-mode "6" for NVME boot, but looks like this eeprom you just gave me do not support it.
SIG pieeprom.sig b440356d2b26f3b6434ea2750239d7e8e239023dea95447092de1297549e5959 1637330510
Reading EEPROM: 524288
Writing EEPROM
++++++++++++++++++++++++++++++..++++++++++++++++++++++++++++++++++++++++++++++++++++++.........................................+
Verify BOOT EEPROM
Reading EEPROM: 524288
BOOT-EEPROM: UPDATED
PM_RSTS: 0x00001000
RPi: BOOTLOADER release VERSION:ca381952 DATE: Nov 19 2021 TIME: 10:22:04 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1637317322 0x72d970ff 0x00d03140 0x000b33c8
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200
PCI reset
PCI reset
USB xHC init failed
Boot mode: SD (01) order f6
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: (06) order f
Unknown boot mode 0x6. Trying SD.
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
So sorry to waste your time. I picked an old branch. Here's the right one, and I've actually tested it! It should show up as version e0c6d67c pieeprom.original3.zip
Okay, that one works better...much better :-) No problem with a reboot at all.
But this is a different brand of NVMe card. The other one is already on it's way back to the vendor. Not sure it I get a replacement or a refund.
SIG pieeprom.sig 09c6c294f0270e91b5d4f591c83090c27bc18fb1a421d9027fd2a252a86d053b 1637336099
Reading EEPROM: 524288
Writing EEPROM
++++++++++++++++++++++++++++++.................................................................................................+
Verify BOOT EEPROM
Reading EEPROM: 524288
BOOT-EEPROM: UPDATED
PM_RSTS: 0x00001000
RPi: BOOTLOADER release VERSION:e0c6d67c DATE: Nov 19 2021 TIME: 14:17:52 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1637331471 0x72d970ff 0x00d03140 0x000c6c09
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200
Boot mode: SD (01) order f6
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: NVME (06) order f
VID 0x1e0f MN KBG40ZNS512G NVMe KIOXIA 512GB
NVME on
MBR: 0x00002000, 524288 type: 0x0c
MBR: 0x00082000,124203008 type: 0x83
MBR: 0x076f500c,875479716 type: 0x83
MBR: 0x00000000, 0 type: 0x00
Trying partition: 0
lba: 8192 oem: 'mkfs.fat' volume: ' boot '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0 r-entries 0
FAT32 clusters 516190
PM_RSTS: 0x00001000
Trying partition: 0
lba: 8192 oem: 'mkfs.fat' volume: ' boot '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0 r-entries 0
FAT32 clusters 516190
Read config.txt bytes 2090 hnd 0x00000000
Read start4.elf bytes 2241504 hnd 0x00000000
Read fixup4.dat bytes 5411 hnd 0x00000000
Firmware: b8a114e5a9877e91ca8f26d1a5ce904b2ad3cf13 Oct 29 2021 10:47:33
0x00d03140 0x00000000 0x00000fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.63-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1459 SMP Wed Oct 6 16:41:57 BST 2021
[ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[ 0.000000] random: fast init done
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff]
[ 0.000000] Normal empty
[ 0.000000] HighMem [mem 0x0000000030000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] percpu: Embedded 20 pages/cpu s50828 r8192 d22900 u81920
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 2059584
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:62:3B:5C vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 root=PARTUUID=18fa53af-02 rootfstype=ext4 fsck.repair=yes rootwait
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x0000000012400000-0x0000000016400000] (64MB)
[ 0.000000] Memory: 7758572K/8245248K available (10240K kernel code, 1364K rwdata, 3192K rodata, 2048K init, 891K bss, 158996K reserved, 327680K cma-reserved, 7458816K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] ftrace: allocating 34315 entries in 68 pages
[ 0.000000] ftrace: allocated 68 pages with 2 groups
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 25)
[ 0.000000] random: get_random_bytes called from start_kernel+0x3c8/0x59c with crng_init=1
[ 0.000008] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[ 0.000034] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[ 0.000094] bcm2835: system timer (irq = 26)
[ 0.000747] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[ 0.000768] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[ 0.000792] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[ 0.000810] Switching to timer-based delay loop, resolution 18ns
[...]
[ 4.594590] systemd[1]: Mounted Kernel Configuration File System.
[ 4.609077] systemd[1]: Finished File System Check on Root Device.
[ 4.621849] systemd[1]: Starting Remount Root and Kernel File Systems...
Raspbian GNU/Linux 11 raspberrypi ttyS0
raspberrypi login: pi
Password:
Linux raspberrypi 5.10.63-v7l+ #1459 SMP Wed Oct 6 16:41:57 BST 2021 armv7l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Nov 19 14:23:20 GMT 2021 on ttyS0
pi@raspberrypi:~$ reboot now
Failed to write reboot parameter file: Permission denied
pi@raspberrypi:~$ sudo su
root@raspberrypi:/home/pi# reboot now
PM_RSTS: 0x00001020
RPi: BOOTLOADER release VERSION:e0c6d67c DATE: Nov 19 2021 TIME: 14:17:52 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1637331471 0x72d970ff 0x00d03140 0x000c657d
PM_RSTS: 0x00001020
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200
Boot mode: SD (01) order f6
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: NVME (06) order f
VID 0x1e0f MN KBG40ZNS512G NVMe KIOXIA 512GB
NVME on
MBR: 0x00002000, 524288 type: 0x0c
MBR: 0x00082000,124203008 type: 0x83
MBR: 0x076f500c,875479716 type: 0x83
MBR: 0x00000000, 0 type: 0x00
Trying partition: 0
lba: 8192 oem: 'mkfs.fat' volume: ' boot '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0 r-entries 0
FAT32 clusters 516190
PM_RSTS: 0x00001020
Trying partition: 0
lba: 8192 oem: 'mkfs.fat' volume: ' boot '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0 r-entries 0
FAT32 clusters 516190
Read config.txt bytes 2090 hnd 0x00000000
Read start4.elf bytes 2241504 hnd 0x00000000
Read fixup4.dat bytes 5411 hnd 0x00000000
Firmware: b8a114e5a9877e91ca8f26d1a5ce904b2ad3cf13 Oct 29 2021 10:47:33
0x00d03140 0x00000000 0x00000fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.63-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1459 SMP Wed Oct 6 16:41:57 BST 2021
[ 0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[ 0.000000] random: fast init done
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff]
[ 0.000000] Normal empty
[ 0.000000] HighMem [mem 0x0000000030000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[...]
``
Thanks, much appreciated. I'll keep this open until a fix is released.
The fix for this should be in released now.
Mandatory information
Describe the bug The system boots without any trouble from the attached NVMe disk. No problem in running the machine, no hiccups, everything works fine. However, a simple "reboot now" got stuck which the following error on the boot console:
This loops endless. The only way to restart the machine is by power-cycling.
Bootloader version and configuration Please include the bootloader version and config.
NVMe boot (please complete the following information):
Additional context
A dump of the successful boot log when turning on the power: