search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.11k stars 4.97k forks source link

M.2 HAT+ - incompatible NVMe (Sandisk Corp WD Blue SN550 NVMe SSD/SanDisk Ultra 3D NVMe 500GB) #6195

Closed aBUGSworstnightmare-rpi closed 3 months ago

aBUGSworstnightmare-rpi commented 4 months ago

Describe the bug

The NVMe is working on CM4 but not even detected on the official M.2 HAT+. Drive is capable of gen3 --> test conducted on gen1/gen2/gen3.

Steps to reproduce the behaviour

Test the NVMe in CM4 (see logs) - NVMe linked on the bus, fully working.

Connect the NVMe to the official M.2 HAT+ --> device is not even detected on the PCIe bus.

Device (s)

Raspberry Pi 5, Raspberry Pi CM4

System

for CM4: Linux raspberrypi 6.6.31-v8+ #1 SMP PREEMPT Fri May 24 14:09:25 CEST 2024 aarch64 GNU/Linux

for Pi5: Linux m2hattest 6.6.31-v8-16k+ #1766 SMP PREEMPT Fri May 24 12:15:34 BST 2024 aarch64 GNU/Linux

Logs

Relevant info from the CM4 (CM4 on CM4IO) On CM4

pi@raspberrypi:~ $ lspci -n 00:00.0 0604: 14e4:2711 (rev 20) 01:00.0 0108: 15b7:5009 (rev 01)

pi@raspberrypi:~ $ sudo lspci -q 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) 01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)

pi@raspberrypi:~ $ sudo lspci -vvxxx 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) (prog-if 00 [Normal decode]) Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 27 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 Memory behind bridge: 00000000-000fffff [size=1M] [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [48] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1 TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+ RootCap: CRSVisible+ RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+ RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+ LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported, DRS- DownstreamComp: Link Down - Not Determined Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 RootCmd: CERptEn+ NFERptEn+ FERptEn+ RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd- FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000 Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?> Capabilities: [240 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=8us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=32us LTR1.2_Threshold=48128ns L1SubCtl2: T_PwrOn=10us Kernel driver in use: pcieport 00: e4 14 11 27 06 00 10 00 20 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00 20: 00 c0 00 c0 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 48 00 00 00 00 00 00 00 1b 01 00 00 40: 00 00 00 00 00 00 00 00 01 ac 13 48 08 20 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 10 00 42 00 b0: 02 80 00 00 10 2c 00 00 12 cc 64 00 42 00 12 d0 c0: 00 00 00 00 00 00 40 00 18 00 01 00 00 00 00 00 d0: 1f 08 08 00 00 04 00 00 06 00 00 80 02 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express]) Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 27 Region 0: Memory at 600000000 (64-bit, non-prefetchable) [size=16K] Region 4: Memory at 600004000 (64-bit, non-prefetchable) [size=256] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [b0] MSI-X: Enable+ Count=17 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=4 offset=00000000 Capabilities: [c0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [1b8 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [300 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=48128ns L1SubCtl2: T_PwrOn=10us Kernel driver in use: nvme 00: b7 15 09 50 06 04 10 00 01 02 08 01 00 00 00 00 10: 04 00 00 c0 00 00 00 00 00 00 00 00 00 00 00 00 20: 04 40 00 c0 00 00 00 00 00 00 00 00 b7 15 09 50 30: 00 00 00 00 80 00 00 00 00 00 00 00 1b 01 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 01 90 03 00 08 00 00 00 00 00 00 00 00 00 00 00 90: 05 b0 8a 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 11 c0 10 80 00 20 00 00 04 00 00 00 00 00 00 00 c0: 10 00 02 00 02 8f 00 10 1f 28 00 00 43 a8 45 00 d0: 42 01 12 10 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 12 08 10 00 00 04 00 00 0e 00 00 00 f0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00

pi@raspberrypi:~ $ uname -a Linux raspberrypi 6.6.31-v8+ #1 SMP PREEMPT Fri May 24 14:09:25 CEST 2024 aarch64 GNU/Linux

pi@raspberrypi:~ $ cat /proc/cpuinfo | grep Revision Revision : c03140

pi@raspberrypi:~ $ lspci 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) 01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01) pi@raspberrypi:~ $ sudo lspci -vvv | grep -E 'nvme|LnkSta' LnkSta: Speed 5GT/s, Width x1 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded) LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- Kernel driver in use: nvme pi@raspberrypi:~ $ sudo nvme id-ctrl -H /dev/nvme0 NVME Identify Controller: vid : 0x15b7 ssvid : 0x15b7 sn : 20511G803175
mn : SanDisk Ultra 3D NVMe
fr : 211070WD rab : 4 ieee : 001b44 cmic : 0 [3:3] : 0 ANA not supported [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port

mdts : 7 cntlid : 0x1 ver : 0x10400 rtd3r : 0x7a120 rtd3e : 0xf4240 oaes : 0x200 [31:31] : 0 Discovery Log Change Notice Not Supported [27:27] : 0 Zone Descriptor Changed Notices Not Supported [15:15] : 0 Normal NSS Shutdown Event Not Supported [14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported [13:13] : 0 LBA Status Information Notices Not Supported [12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported [11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported [9:9] : 0x1 Firmware Activation Notices Supported [8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0x2 [19:19] : 0 Flexible Data Placement Not Supported [15:15] : 0 Extended LBA Formats Not Supported [14:14] : 0 Delete NVM Set Not Supported [13:13] : 0 Delete Endurance Group Not Supported [12:12] : 0 Variable Capacity Management Not Supported [11:11] : 0 Fixed Capacity Management Not Supported [10:10] : 0 Multi Domain Subsystem Not Supported [9:9] : 0 UUID List Not Supported [8:8] : 0 SQ Associations Not Supported [7:7] : 0 Namespace Granularity Not Supported [6:6] : 0 Traffic Based Keep Alive Not Supported [5:5] : 0 Predictable Latency Mode Not Supported [4:4] : 0 Endurance Groups Not Supported [3:3] : 0 Read Recovery Levels Not Supported [2:2] : 0 NVM Sets Not Supported [1:1] : 0x1 Non-Operational Power State Permissive Supported [0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0 cntrltype : 1 [7:2] : 0 Reserved [1:0] : 0x1 I/O Controller fguid : 00000000-0000-0000-0000-000000000000 crdt1 : 0 crdt2 : 0 crdt3 : 0 nvmsr : 0 [1:1] : 0 NVM subsystem Not part of an Enclosure [0:0] : 0 NVM subsystem Not part of an Storage Device

vwci : 0 [7:7] : 0 VPD Write Cycles Remaining field is Not valid. [6:0] : 0 VPD Write Cycles Remaining

mec : 0 [1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port [0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port

oacs : 0x17 [10:10] : 0 Lockdown Command and Feature Not Supported [9:9] : 0 Get LBA Status Capability Not Supported [8:8] : 0 Doorbell Buffer Config Not Supported [7:7] : 0 Virtualization Management Not Supported [6:6] : 0 NVMe-MI Send and Receive Not Supported [5:5] : 0 Directives Not Supported [4:4] : 0x1 Device Self-test Supported [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Security Send and Receive Supported

acl : 4 aerl : 7 frmw : 0x14 [5:5] : 0 Multiple FW or Boot Update Detection Not Supported [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x2 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0x1e [6:6] : 0 Telemetry Log Data Area 4 Not Supported [5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported [4:4] : 0x1 Persistent Event log Supported [3:3] : 0x1 Telemetry host/controller initiated log page Supported [2:2] : 0x1 Extended data for Get Log Page Supported [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 255 [7:0] : 255 (0's based) Error Log Page Entries (ELPE)

npss : 4 [7:0] : 4 (0's based) Number of Power States Support (NPSS)

avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format

apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported

wctemp : 353 [15:0] : 80°C (353 Kelvin) Warning Composite Temperature Threshold (WCTEMP)

cctemp : 358 [15:0] : 85°C (358 Kelvin) Critical Composite Temperature Threshold (CCTEMP)

mtfa : 50 hmpre : 51200 hmmin : 823 tnvmcap : 500,107,862,016 [127:0] : 500,107,862,016 Total NVM Capacity (TNVMCAP)

unvmcap : 0 [127:0] : 0 Unallocated NVM Capacity (UNVMCAP)

rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units

edstt : 57 dsto : 1 fwug : 1 kas : 0 hctma : 0x1 [0:0] : 0x1 Host Controlled Thermal Management Supported

mntmt : 273 [15:0] : 0°C (273 Kelvin) Minimum Thermal Management Temperature (MNTMT)

mxtmt : 358 [15:0] : 85°C (358 Kelvin) Maximum Thermal Management Temperature (MXTMT)

sanicap : 0x60000002 [31:30] : 0x1 Media is not additionally modified after sanitize operation completes successfully [29:29] : 0x1 No-Deallocate After Sanitize bit in Sanitize command Not Supported [2:2] : 0 Overwrite Sanitize Operation Not Supported [1:1] : 0x1 Block Erase Sanitize Operation Supported [0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 0 hmmaxd : 8 nsetidmax : 0 endgidmax : 0 anatt : 0 anacap : 0 [7:7] : 0 Non-zero group ID Not Supported [6:6] : 0 Group ID does change [4:4] : 0 ANA Change state Not Supported [3:3] : 0 ANA Persistent Loss state Not Supported [2:2] : 0 ANA Inaccessible state Not Supported [1:1] : 0 ANA Non-optimized state Not Supported [0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0 nanagrpid : 0 pels : 1 domainid : 0 megcap : 0 sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 0 nn : 1 oncs : 0x5f [8:8] : 0 Copy Not Supported [7:7] : 0 Verify Not Supported [6:6] : 0x1 Timestamp Supported [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0x1 Compare Supported

fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported

fna : 0 [3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported [2:2] : 0 Crypto Erase Not Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s)

vwc : 0x7 [2:1] : 0x3 The Flush command supports NSID set to FFFFFFFFh [0:0] : 0x1 Volatile Write Cache Present

awun : 0 awupf : 0 icsvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format

nwpc : 0 [2:2] : 0 Permanent Write Protect Not Supported [1:1] : 0 Write Protect Until Power Supply Not Supported [0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 0 ocfs : 0 [1:1] : 0 Controller Copy Format 1h Not Supported [0:0] : 0 Controller Copy Format 0h Not Supported

sgls : 0 [15:8] : 0 SGL Descriptor Threshold [1:0] : 0 Scatter-Gather Lists Not Supported

mnan : 0 maxdna : 0 maxcna : 0 subnqn : nqn.2018-01.com.wdc:nguid:E8238FA6BF53-0001-001B448B484C36E0 ioccsz : 0 iorcsz : 0 icdoff : 0 fcatt : 0 [0:0] : 0 Dynamic Controller Model

msdbd : 0 ofcs : 0 [0:0] : 0 Disconnect command Not Supported

ps 0 : mp:3.50W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:0.6300W active_power:2.10W active_power_workload:80K 128KiB SW ps 1 : mp:2.40W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:0.6300W active_power:1.60W active_power_workload:80K 128KiB SW ps 2 : mp:1.90W operational enlat:0 exlat:0 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:0.6300W active_power:1.50W active_power_workload:80K 128KiB SW ps 3 : mp:0.0200W non-operational enlat:3900 exlat:11000 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:0.0200W active_power:- active_power_workload:- ps 4 : mp:0.0050W non-operational enlat:5000 exlat:39000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:0.0050W active_power:- active_power_workload:-

When connected to the official HAT+ the NVMe is not detected on th bus, external PCIe is down. pi@m2hattest:~ $ lspci -n 00:00.0 0604: 14e4:2712 (rev 21) 01:00.0 0200: 1de4:0001 pi@m2hattest:~ $ sudo lspci -q 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21) 01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge pi@m2hattest:~ $ sudo lspci -vvxxx 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21) (prog-if 00 [Normal decode]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 39 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 Memory behind bridge: 00000000-005fffff [size=6M] [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [48] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <2us ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+ RootCap: CRSVisible+ RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+ RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+ AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+ LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported, DRS- DownstreamComp: Link Up - Present Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 RootCmd: CERptEn+ NFERptEn+ FERptEn+ RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd- FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000 Capabilities: [160 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?> Capabilities: [240 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=8us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=1us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [300 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Kernel driver in use: pcieport 00: e4 14 12 27 06 00 10 00 21 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00 20: 00 00 50 00 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 48 00 00 00 00 00 00 00 27 01 00 00 40: 00 00 00 00 00 00 00 00 01 ac 13 48 08 20 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 10 00 42 00 b0: 02 80 00 00 30 2c 00 00 42 c8 64 00 42 00 42 d0 c0: 00 00 00 00 00 00 40 00 18 00 01 00 00 00 00 00 d0: 3f 08 08 00 00 00 00 00 06 00 00 80 02 00 00 40 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 39 Region 0: Memory at 1f00410000 (32-bit, non-prefetchable) [size=16K] Region 1: Memory at 1f00000000 (32-bit, non-prefetchable) [virtual] [size=4M] Region 2: Memory at 1f00400000 (32-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [b0] MSI-X: Enable+ Count=61 Masked- Vector table: BAR=0 offset=00000000 PBA: BAR=0 offset=00002000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Kernel driver in use: rp1 00: e4 1d 01 00 06 04 10 00 00 00 00 02 00 00 00 00 10: 00 00 41 00 00 00 00 00 00 00 40 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 27 01 00 00 40: 01 70 c3 db 08 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 10 b0 02 00 c1 8f 00 00 3f 20 11 00 42 cc 40 00 80: 42 00 42 10 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 10 00 00 00 00 00 00 00 06 00 00 00 a0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 11 00 3c 80 00 00 00 00 00 20 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

pi@m2hattest:~ $ uname -a Linux m2hattest 6.6.31-v8-16k+ #1766 SMP PREEMPT Fri May 24 12:15:34 BST 2024 aarch64 GNU/Linux pi@m2hattest:~ $ cat /proc/cpuinfo | grep Revision Revision : d04170 pi@m2hattest:~ $ lspci 00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 01:00.0 Ethernet controller: Device 1de4:0001 pi@m2hattest:~ $ sudo lspci -vvv | grep -E 'nvme|LnkSta' LnkSta: Speed 5GT/s, Width x4 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- LnkSta: Speed 5GT/s, Width x4 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- pi@m2hattest:~ $ sudo nvme id-ctrl -H /dev/nvme0 /dev/nvme0: No such file or directory Usage: nvme id-ctrl [OPTIONS]

Send an Identify Controller command to the given device and report information about the specified controller in human-readable or binary format. May also return vendor-specific controller attributes in hex-dump if requested.

Options: [ --vendor-specific, -v ] --- dump binary vendor field [ --output-format=, -o ] --- Output format: normal|json|binary [ --raw-binary, -b ] --- show identify in binary format [ --human-readable, -H ] --- show identify in readable format pi@m2hattest:~ $

Tested with PCIe_gen forced to 1 (param=pciex1_gen=1) raspinfo output: https://paste.debian.net/hidden/d5bb727e/

Tested with PCIe_gen forced to 2 param=pciex1_gen=2) https://paste.debian.net/hidden/950b4ce8/

as the drive is capable tested at gen3 as well --> all non working https://paste.debian.net/hidden/d19c0467/

Additional context

No response

pelwell commented 4 months ago

For testing purposes, does dtparam=nvme in config.txt make any difference?

aBUGSworstnightmare-rpi commented 4 months ago

For testing purposes, does dtparam=nvme in config.txt make any difference?

Adding the parameter shows no effect; external PCIe is still down. https://paste.debian.net/hidden/44e00785/

to my current understanding 'PCIE_PROBE=1' in EEPROM config is the same. Tested as well, but link remains down.

timg236 commented 4 months ago

The boot UART log will indicate the status the DET WAKE signal and PCIE_PROBE will force PWR_EN.

It should be possible to see the state of those pins via pinctrl

aBUGSworstnightmare-rpi commented 4 months ago

The boot UART log will indicate the status the DET WAKE signal and PCIE_PROBE will force PWR_EN.

It should be possible to see the state of those pins via pinctrl

Find the log below. I've used a Pi Debug probe and 'minicom' (on another Pi5) for caputring the boot loader output.

3.18 PCIEx1: PWR 0 DET_WAKE 1

Question: why does the boot loader provides info of 4GB RAM while the Pi is an 8GB version pi@m2hattest:~ $ vcgencmd get_config total_mem total_mem=8192 or how to read

1.35 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267 1.38 DDR 4267 1 0 64 152

0.82 RPi: BOOTSYS release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:29:43 0.86 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945383 serial 9a8412f7 boardrev d04170 stc 886147 0.96 AON_RESET: 00000003 PM_RSTS 00001000 0.03 RP1_BOOT chip ID: 0x20001927 0.06 PM_RSTS: 0x00001000 0.06 part 00000000 reset_info 00000000 0.10 PMIC reset-event 00000000 rtc 6658738f alarm 00000000 enabled 0 0.16 uSD voltage 3.3V 1.35 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267 1.38 DDR 4267 1 0 64 152 2.55 OTP boardrev d04170 bootrom a a 2.56 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000 2.63 VC-JTAG unlocked 2.86 RP1_BOOT chip ID: 0x20001927

3.93 RP1_BOOT chip ID: 0x20001927 3.94 RP1_BOOT: fw size 25992 3.49 PCI2 init 3.49 PCI2 reset 3.93 PCIe scan 00001de4:00000001 3.94 RP1_CHIP_INFO 20001927

3.97 RPi: BOOTLOADER release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:29:43 3.04 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945383 serial 9a8412f7 boardrev d04170 stc 3904199 3.14 AON_RESET: 00000003 PM_RSTS 00001000 3.18 PCIEx1: PWR 0 DET_WAKE 1 3.45 usb_pd_init status 5 3.51 XHCI-STOP 3.51 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.55 USBSTS 1 3.58 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.63 xHC0 ports 3 slots 64 intrs 4 3.75 XHCI-STOP 3.75 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.79 USBSTS 1 3.82 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.87 xHC1 ports 3 slots 64 intrs 4 3.94 Boot mode: SD (01) order f46 4.09 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 4.20 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 4.50 OCR c0ff8000 [190] CID: 000353445343333247809318a100014b CSD: 400e00325b590000edc87f800a404000 4.59 SD: bus-width: 4 spec: 2 SCR: 0x02358043 0x00000000 4.66 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 4.78 MBR: 0x00002000, 1048576 type: 0x0c 4.79 MBR: 0x00102000,61277184 type: 0x83 4.83 MBR: 0x00000000, 0 type: 0x00 4.87 MBR: 0x00000000, 0 type: 0x00 4.91 Trying partition: 0 4.96 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 261116 (4) 4.00 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0 4.08 FAT32 clusters 261116 4.14 [sdcard] autoboot.txt not found 4.15 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0 4.22 Trying partition: 0 4.26 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 261115 (4) 4.30 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0 4.39 FAT32 clusters 261115 4.43 Read config.txt bytes 2717 hnd 0x2808e 4.47 [sdcard] pieeprom.upd not found 4.48 usb_max_current_enable default 0 max-current 5000 4.60 Read bcm2712-rpi-5-b.dtb bytes 80218 hnd 0x1defd 4.62 dt-match: compatible: raspberrypi,5-model-b match: brcm,bcm2712 4.69 dt-match: compatible: brcm,bcm2712 match: brcm,bcm2712

NOTICE: BL31: v2.6(release):v2.6-239-g2a9ede0bd NOTICE: BL31: Built : 14:26:57, Jun 22 2023 [ 0.839738] brcm-pcie 1000110000.pcie: link down

Debian GNU/Linux 12 m2hattest ttyAMA10

m2hattest login:

timg236 commented 4 months ago

It doesn't say 4GB. Your log and the HDMI diagnostic screen will say it's a 64 GBit so 8 gigabyte SDRAM "1.35 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267"

timg236 commented 4 months ago

Regarding the original question - "3.18 PCIEx1: PWR 0 DET_WAKE 1"

So the M2.HAT is detected and the bootloader will have enabled power to the NVMe and enabled the PCIe as a bootable device. This particular boot had SD has a higher priority so it's not possible to determine whether bootloader could see the NVMe device, but probably not if the device is not visible under Linux.

Do you see the amber power indicator on the M2.HAT ?

aBUGSworstnightmare-rpi commented 4 months ago

Regarding the original question - "3.18 PCIEx1: PWR 0 DET_WAKE 1"

So the M2.HAT is detected and the bootloader will have enabled power to the NVMe and enabled the PCIe as a bootable device. This particular boot had SD has a higher priority so it's not possible to determine whether bootloader could see the NVMe device, but probably not if the device is not visible under Linux.

Do you see the amber power indicator on the M2.HAT ?

the amber PWR LED is on.

changed the boot order to 0f416

0.81 RPi: BOOTSYS release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:29:43 0.85 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945383 serial 9a8412f7 boardrev d04170 stc 885883 0.95 AON_RESET: 00000003 PM_RSTS 00001000 0.03 RP1_BOOT chip ID: 0x20001927 0.05 PM_RSTS: 0x00001000 0.06 part 00000000 reset_info 00000000 0.09 PMIC reset-event 00000000 rtc 6658b261 alarm 00000000 enabled 0 0.16 uSD voltage 3.3V 1.35 Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267 1.38 DDR 4267 1 0 64 152 2.57 OTP boardrev d04170 bootrom a a 2.59 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000 2.66 VC-JTAG unlocked 2.89 RP1_BOOT chip ID: 0x20001927

3.96 RP1_BOOT chip ID: 0x20001927 3.96 RP1_BOOT: fw size 25992 3.51 PCI2 init 3.51 PCI2 reset 3.96 PCIe scan 00001de4:00000001 3.96 RP1_CHIP_INFO 20001927

3.99 RPi: BOOTLOADER release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:29:43 3.06 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945383 serial 9a8412f7 boardrev d04170 stc 3906607 3.16 AON_RESET: 00000003 PM_RSTS 00001000 3.20 PCIEx1: PWR 0 DET_WAKE 1 3.49 usb_pd_init status 5 3.53 XHCI-STOP 3.53 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.58 USBSTS 11 3.60 xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.65 xHC0 ports 3 slots 64 intrs 4 3.77 XHCI-STOP 3.77 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.82 USBSTS 11 3.84 xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d 3.89 xHC1 ports 3 slots 64 intrs 4 3.97 Boot mode: NVME (06) order f41 4.98 USB2[2] 000206e1 connected 4.13 USB3[3] 002a1203 connected enabled 4.14 USB3 root HUB port 3 init 4.18 DEV [01:00] 3.00 000000:03 class 0 VID 05dc PID b051 4.22 MSD device [01:00] 3.00 000000:03 conf 0 iface 0 ep 81#1024 02#1024 4.29 MSD [01:00] 3.00 000000:03 register MSD 4.48 USB2[2] 00200e03 connected enabled 4.49 USB2 root HUB port 2 init 4.54 DEV [01:00] 2.00 000000:02 class 9 VID 05e3 PID 0610 4.57 HUB init [01:00] 2.00 000000:02 4.70 HUB [01:00] 2.00 000000:02 init port 1 speed 2 4.99 DEV [02:01] 2.00 000001:02 class 0 VID 04d9 PID 0006 4.06 HID [02:01] 2.00 000001:02 register HID 4.76 HUB [01:00] 2.00 000000:02 init port 3 speed 2 4.82 DEV [03:01] 1.16 000003:02 class 0 VID 093a PID 2510 4.87 HID [03:01] 1.16 000003:02 register HID 4.17 PCI1 init 4.17 PCI1 reset 5.38 PCIe timeout: 0x0001e08f 5.38 Failed to open device: 'nvme' 5.41 Retry NVME 1 5.43 PCI1 init 5.44 PCI1 reset 5.66 PCIe timeout: 0x0001e08f 5.67 Failed to open device: 'nvme' 5.70 Boot mode: SD (01) order f4 5.73 HUB [01:00] 2.00 000000:02 init port 4 speed 3 5.92 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 5.03 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 5.19 DEV [04:01] 2.00 000004:02 class 0 VID 148f PID 5572 5.34 OCR c0ff8000 [188] CID: 000353445343333247809318a100014b CSD: 400e00325b590000edc87f800a404000 5.44 SD: bus-width: 4 spec: 2 SCR: 0x02358043 0x00000000 5.51 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 5.63 MBR: 0x00002000, 1048576 type: 0x0c 5.64 MBR: 0x00102000,61277184 type: 0x83 5.67 MBR: 0x00000000, 0 type: 0x00 5.71 MBR: 0x00000000, 0 type: 0x00 5.75 Trying partition: 0 5.80 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 261116 (4) 5.84 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0 5.92 FAT32 clusters 261116 5.99 [sdcard] autoboot.txt not found 5.00 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0 5.06 Trying partition: 0 5.11 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 261115 (4) 5.15 rsc 32 fat-sectors 2040 root dir cluster 2 sectors 0 entries 0 5.24 FAT32 clusters 261115 5.28 Read config.txt bytes 2717 hnd 0x2808e 5.32 [sdcard] pieeprom.upd not found 5.33 usb_max_current_enable default 0 max-current 5000 5.45 Read bcm2712-rpi-5-b.dtb bytes 80218 hnd 0x1defd 5.47 dt-match: compatible: raspberrypi,5-model-b match: brcm,bcm2712 5.53 dt-match: compatible: brcm,bcm2712 match: brcm,bcm2712

NOTICE: BL31: v2.6(release):v2.6-239-g2a9ede0bd NOTICE: BL31: Built : 14:26:57, Jun 22 2023 [ 0.839764] brcm-pcie 1000110000.pcie: link down

aBUGSworstnightmare-rpi commented 4 months ago

Any idea what is causing the link to time out? Any parameter which I can add to prevent this from happening?

What is different when comparing PCIe on CM4 to PCIe on Pi5?

aBUGSworstnightmare-rpi commented 4 months ago

As no-one seems to be able to answer what is the difference between PCIe on CM4 (working SN550) compared to Pi5 (SN550 non-working) let me add some CM4 bootlaoder logs

booted with no PCIe NVMe drive attached

0.10 RPi: BOOTSYS release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:26:58 0.14 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945218 serial 6906841c boardrev c03140 stc 514243 0.30 PM_RSTS: 0x00001020 0.30 part 00000000 reset_info 00000000 0.34 uSD voltage 3.3V 0.59 Initialising SDRAM 'Samsung' 16Gb x2 total-size: 32 Gbit 3200 0.63 DDR 3200 1 0 32 152 2.98 OTP boardrev c03140 bootrom 48b0 48b0 2.00 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000 2.07 VC-JTAG unlocked

2.83 RPi: BOOTLOADER release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:26:58 2.87 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945218 serial 6906841c boardrev c03140 stc 2387769 2.23 PCI0 init 2.23 PCI0 reset 2.43 PCIe timeout: 0x0000008f 2.43 USB xHC init failed 2.46 Boot mode: SD (01) order f2546 2.63 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 2.74 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 3.90 OCR c0ff8000 [261] CID: 00035344534c363447804a6e0d1a0114 CSD: 400e00325b590001dbd37f800a404000 3.01 SD: bus-width: 4 spec: 2 SCR: 0x02458003 0x00000000 3.09 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 3.22 MBR: 0x00002000, 1048576 type: 0x0c 3.23 MBR: 0x00102000,123678720 type: 0x83 3.27 MBR: 0x00000000, 0 type: 0x00 3.31 MBR: 0x00000000, 0 type: 0x00 3.34 Trying partition: 0 3.40 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 130812 (8) 3.43 rsc 32 fat-sectors 1024 root dir cluster 2 sectors 0 entries 0 3.52 FAT32 clusters 130812 3.59 [sdcard] autoboot.txt not found 3.60 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0 3.67 Trying partition: 0 3.70 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 130812 (8) 3.75 rsc 32 fat-sectors 1024 root dir cluster 2 sectors 0 entries 0 3.85 FAT32 clusters 130812 3.87 Read config.txt bytes 3431 hnd 0xcf85 3.91 [sdcard] pieeprom.upd not found 3.93 [sdcard] recover4.elf not found 3.97 [sdcard] recovery.elf not found 3.20 Read start4.elf bytes 2257216 hnd 0xaa26 3.23 Read fixup4.dat bytes 5436 hnd 0xbaf5 3.26 0x00c03140 0x00000000 0x00001fff 3.30 MEM GPU: 76 ARM: 948 TOTAL: 1024 3.08 Firmware: 4942b7633c0ff1af1ee95a51a33b56a9dae47529 May 24 2024 15:30:04 3.41 Starting start4.elf @ 0xfec00200 partition 0 3.43 PCI0 reset 3.55 +

0.10 RPi: BOOTSYS release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:26:58 0.14 BOOTMODE: 0x06 partition 63 build-ts BUILD_TIMESTAMP=1715945218 serial 6906841c boardrev c03140 stc 514243 0.30 Halt: wake: 1 power_off: 0 --> powered down

--> attaching the SN550 to PCIe

0.11 RPi: BOOTSYS release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:26:58 0.15 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945218 serial 6906841c boardrev c03140 stc 515918 0.32 PM_RSTS: 0x00001000 0.32 part 00000000 reset_info 00000000 0.35 uSD voltage 3.3V 0.61 Initialising SDRAM 'Samsung' 16Gb x2 total-size: 32 Gbit 3200 0.64 DDR 3200 1 0 32 152 2.01 OTP boardrev c03140 bootrom 48b0 48b0 2.03 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000 2.10 VC-JTAG unlocked

2.86 RPi: BOOTLOADER release VERSION:72caf667 DATE: 2024/05/17 TIME: 12:26:58 2.90 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1715945218 serial 6906841c boardrev c03140 stc 2390617 2.26 PCI0 init 2.26 PCI0 reset 2.29 PCIe scan 000015b7:00005009 2.30 USB xHC init failed 2.32 Boot mode: SD (01) order f2546 2.49 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 2.60 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 2.83 OCR c0ff8000 [269] CID: 00035344534c363447804a6e0d1a0114 CSD: 400e00325b590001dbd37f800a404000 2.93 SD: bus-width: 4 spec: 2 SCR: 0x02458003 0x00000000 2.01 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 2.14 MBR: 0x00002000, 1048576 type: 0x0c 2.15 MBR: 0x00102000,123678720 type: 0x83 2.19 MBR: 0x00000000, 0 type: 0x00 2.23 MBR: 0x00000000, 0 type: 0x00 2.27 Trying partition: 0 2.32 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 130812 (8) 2.36 rsc 32 fat-sectors 1024 root dir cluster 2 sectors 0 entries 0 2.45 FAT32 clusters 130812 2.52 [sdcard] autoboot.txt not found 2.52 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 0 2.59 Trying partition: 0 2.62 type: 32 lba: 8192 'mkfs.fat' ' bootfs ' clusters 130812 (8) 2.68 rsc 32 fat-sectors 1024 root dir cluster 2 sectors 0 entries 0 2.77 FAT32 clusters 130812 2.79 Read config.txt bytes 3431 hnd 0xcf85 2.83 [sdcard] pieeprom.upd not found 2.86 [sdcard] recover4.elf not found 2.89 [sdcard] recovery.elf not found 3.13 Read start4.elf bytes 2257216 hnd 0xaa26 3.16 Read fixup4.dat bytes 5436 hnd 0xbaf5 3.18 0x00c03140 0x00000000 0x00001fff 3.22 MEM GPU: 76 ARM: 948 TOTAL: 1024 3.01 Firmware: 4942b7633c0ff1af1ee95a51a33b56a9dae47529 May 24 2024 15:30:04 3.34 Starting start4.elf @ 0xfec00200 partition 0 3.36 PCI0 reset 3.47 +

pi@raspberrypi:~ $ lspci
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20)
01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)
timg236 commented 4 months ago

Until it works under Linux there's little point in trying to boot from this. There are some assorted PCIe compatibility levers that you could try here dtoverlay=pciex1-compat-pi5,= https://github.com/raspberrypi/firmware/issues/1833#issuecomment-2168427105

aBUGSworstnightmare-rpi commented 4 months ago

Working on CM4 - under Linux - doesn't count or what? What is different from PCIe when comparing CM4 and Pi5? It's non working on Pi5 - hence not an NVMe issue, nor a Linux one!

timg236 commented 4 months ago

It working on a CM4 obviously won’t help you with a pi5

aBUGSworstnightmare-rpi commented 4 months ago

It working on a CM4 obviously won’t help you with a pi5

That's why I'm asking here what makes PCIe on Pi5 differ from PCIe on CM4, but either nobody is able or not willing wo answer.

Whst are these 'compatibility flags' good for and why wasn't there none for CM4 do far?

pelwell commented 4 months ago

Because it's a different version of the PCIe RC IP, Gen-3-capable, etc.

timg236 commented 4 months ago

The question is really, what's different about this drive vs all the other drives that work on Pi5. I've already walked you through the steps to verify that the M2.HAT is working and powering on the drive etc.

The assorted Linux PCIe compatiblity quirks to disable features e.g. ASPM are an obvious place to start when trying to isolate the problem.

aBUGSworstnightmare-rpi commented 4 months ago

The question is really, what's different about this drive vs all the other drives that work on Pi5. I've already walked you through the steps to verify that the M2.HAT is working and powering on the drive etc.

The assorted Linux PCIe compatiblity quirks to disable features e.g. ASPM are an obvious place to start when trying to isolate the problem.

Went through all this permutations and always end up with a down link (yes, I know that first two parameters were superfluous, but as I've previously been requested to check if dtparam=nvme makes a difference I've kept them in)

4.02 Boot mode: NVME (06) order f41 4.02 PCI1 init 4.02 PCI1 reset 5.23 PCIe timeout: 0x0001e08f 5.23 Failed to open device: 'nvme' 5.26 Retry NVME 1 5.28 PCI1 init 5.30 PCI1 reset 5.52 PCIe timeout: 0x0001e08f 5.52 Failed to open device: 'nvme' .. NOTICE: BL31: v2.6(release):v2.6-239-g2a9ede0bd NOTICE: BL31: Built : 14:26:57, Jun 22 2023 [ 0.815752] brcm-pcie 1000110000.pcie: link down

# PCIe related parameters
dtparam=pciex1
dtparam=pciex1_gen=2
#dtoverlay=pciex1-compat-pi5,l1ss
#dtoverlay=pciex1-compat-pi5,no-l0s
#dtoverlay=pciex1-compat-pi5,l1ss,no-l0s
#dtoverlay=pciex1-compat-pi5,no-mip
#dtoverlay=pciex1-compat-pi5,l1ss,no-mip
#dtoverlay=pciex1-compat-pi5,no-l0s,no-mip
#dtoverlay=pciex1-compat-pi5,l1ss,no-l0s,no-mip
#dtoverlay=pciex1-compat-pi5,mmio-hi
#dtoverlay=pciex1-compat-pi5,l1ss,mmio-hi
#dtoverlay=pciex1-compat-pi5,l1ss,no-l0s,mmio-hi
dtoverlay=pciex1-compat-pi5,l1ss,no-l0s,no-mip,mmio-hi

Because it's a different version of the PCIe RC IP, with 2 lanes, Gen-3-capable, etc.

sorry, but do I read/understand that correctly here? External PCIe is 2-lanes, but Pi5 exposes one lane only? Or what is the background of this comment? Under the context of having an external 2-lane PCIe interface the naming of the overlay would make some sense at least (for dealing with quirks when running it single lane only).

pelwell commented 4 months ago

External PCIe is 2-lanes, but Pi5 exposes one lane only?

No it's not - I had a feeling that one of the non-RP1 PCIe RCs was 2-lane capable, but I was remembering wrong and should have checked my facts first. The comment has been corrected to avoid further confusion.

aBUGSworstnightmare-rpi commented 4 months ago

@iamursky This is not related to the topic discussed in this issue most likely. Please open a new issue.

aBUGSworstnightmare-rpi commented 3 months ago

as a result of my findings here https://github.com/raspberrypi/linux/issues/6228 I've ended up with a working NVMe on Pi5 now!

As I already was under the impression that this issue might be related to a firmware issue as well I've started to investigate further down that road. Testbed for what is shown next is the CM4 as the drive is running there!

started checking in detail again what I have in front of me

pi@raspberrypi:~ $ cat /sys/class/nvme/nvme0/model && cat /sys/class/nvme/nvme0/firmware_rev
SanDisk Ultra 3D NVMe                   
211070WD

checking based on https://wddashboarddownloads.wdc.com/wdDashboard/config/devices/lista_devices.xml reveals

<!-- Calypso - WD Blue and SanDisk Ultra -->
<lista_device model="SanDisk Ultra 3D NVMe">
<url>
 wdDashboard/firmware/SanDisk_Ultra_3D_NVMe/211210WD/device_properties.xml
</url>

NOTE: WHAT IS SHOWN NEXT IS NOT OFFICIALLY SUPPORTED BY WD --> USE AT OWN RISK!!!

checking the decency of that firmware shows it should be compatible (as it depends on 211070WD which is what my drive is using) 

pi@raspberrypi:~ $ curl https://wddashboarddownloads.wdc.com/wdDashboard/firmware/SanDisk_Ultra_3D_NVMe/211210WD/device_properties.xml

<?xml version="1.0"?>
<ffu version="1.0">
    <model>SanDisk Ultra 3D NVMe</model>
    <fwmode>mode3</fwmode>
    <fwversion>211210WD</fwversion>
    <fwfile>211210WD.fluf</fwfile>
    <dependency model="SanDisk Ultra 3D NVMe">211070WD</dependency>
</ffu>

the new firmware is '211210WD.fluf' so let's download it

pi@raspberrypi:~ $ curl --remote-name https://wddashboarddownloads.wdc.com/wdDashboard/firmware/SanDisk_Ultra_3D_NVMe_500GB/211210WD/211210WD.fluf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   342    0   342    0     0    428      0 --:--:-- --:--:-- --:--:--   428

hmm .. only 342kb? looks strange but let's give it a try.

as firmware can be managed by name-cli I've started the process. Refer to nvmi-cli documentation for details on the commands in use.

pi@raspberrypi:~ $ sudo nvme fw-log /dev/nvme0
Firmware Log for device:nvme0
afi  : 0x1
frs1 : 0x4457303730313132 (211070WD)
pi@raspberrypi:~ $ sudo nvme id-ctrl /dev/nvme0 -H | grep Firmware
  [9:9] : 0x1   Firmware Activation Notices Supported
  [4:4] : 0x1   Firmware Activate Without Reset Supported
  [3:1] : 0x2   Number of Firmware Slots
  [0:0] : 0 Firmware Slot 1 Read/Write

pi@raspberrypi:~ $ sudo nvme fw-download -f 211210WD.fluf /dev/nvme0
Invalid size:342 for f/w image

BAM! That firmware file seems to be corrupted as it is reported as too small. DEAD END?

No! As I also have a SN550 in my possession (and lspci reports it as such anyhow - see https://github.com/raspberrypi/linux/issues/6190) I took a closer look to that one again as well:

pi@raspberrypi:~ $ cat /sys/class/nvme/nvme0/model && cat /sys/class/nvme/nvme0/firmware_rev
WDC WDS500G2B0C-00PXH0                  
233010WD

so let's go back to https://wddashboarddownloads.wdc.com/wdDashboard/config/devices/lista_devices.xml and look for that model and what WDdshboard would list

<!-- Calypso - WD Blue and SanDisk Ultra -->
<lista_device model="WDC WDS250G2B0C-00PXH0">
<url>
wdDashboard/firmware/WDC_WDS250G2B0C-00PXH0/211210WD/device_properties.xml
</url>
</lista_device>

o.k. let's check that firmware dependency

pi@raspberrypi:~ $ curl https://wddashboarddownloads.wdc.com/wdDashboard/firmware/WDC_WDS500G2B0C-00PXH0/211210WD/device_properties.xml

<?xml version="1.0"?>
<ffu version="1.0">
    <model>WDC WDS500G2B0C-00PXH0</model>
    <fwmode>mode3</fwmode>
    <fwversion>211210WD</fwversion>
    <fwfile>211210WD.fluf</fwfile>
    <dependency model="WDC WDS500G2B0C-00PXH0">211070WD</dependency>
</ffu>

BINGO! Shows it depends on 211070WD which is exactly the FW of my Sandisk branded NVMe.

So get that one and try flashing it

pi@raspberrypi:~ $ curl --remote-name https://wddashboarddownloads.wdc.com/wdDashboard/firmware/WDC_WDS500G2B0C-00PXH0/211210WD/211210WD.flufD.fluf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2528k  100 2528k    0     0  1489k      0  0:00:01  0:00:01 --:--:-- 1489k

pi@raspberrypi:~ $ sudo nvme fw-download -f 211210WD.fluf /dev/nvme0
Firmware download success

pi@raspberrypi:~ $ sudo nvme fw-commit -s 2 -a 0 /dev/nvme0
Success committing firmware action:0 slot:2
Multiple Update Detected (MUD) Value: 2457155
Detected an overlapping firmware/boot partition image update command sequence due to processing a command from a Management EndpointDetected an overlapping firmware/boot partitinvme fw-log /dev/nvme0d sequence due to processing a command from an Admin SQ on a controller

pi@raspberrypi:~ $ sudo nvme fw-log /dev/nvme0
Firmware Log for device:nvme0
afi  : 0x1
frs1 : 0x4457303730313132 (211070WD)
frs2 : 0x4457303132313132 (211210WD)

pi@raspberrypi:~ $ sudo nvme fw-commit -s 2 -a 2 /dev/nvme0
Success committing firmware action:2 slot:2
Multiple Update Detected (MUD) Value: 37
Detected an overlapping firmware/boot partition image update command sequence due to processing a command from a Management Endpoint

although 'Firmware Activate Without Reset' is marked as supported I've manually triggered a Reset sudo nvme reset /dev/nvme0

after the reboot re-checking the drive shows that is it now running another firmware

pi@raspberrypi:~ $ lspci
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20)
01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)
pi@raspberrypi:~ $ sudo lspci -vvxxx
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) (prog-if 00 [Normal decode])
    Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 27
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    Memory behind bridge: 00000000-000fffff [size=1M] [32-bit]
    Prefetchable memory behind bridge: [disabled] [64-bit]
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: [48] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
        DevCap: MaxPayload 512 bytes, PhantFunc 0
            ExtTag- RBE+
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
            ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
        LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x1
            TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
        RootCap: CRSVisible+
        RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
        RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
             AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, ARIFwd-
             AtomicOpsCtl: ReqEn- EgressBlck-
        LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported, DRS-
             DownstreamComp: Link Down - Not Determined
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
        RootCmd: CERptEn+ NFERptEn+ FERptEn+
        RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
             FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
        ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
    Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
    Capabilities: [240 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=32us LTR1.2_Threshold=48128ns
        L1SubCtl2: T_PwrOn=10us
    Kernel driver in use: pcieport
00: e4 14 11 27 06 00 10 00 20 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 00
20: 00 c0 00 c0 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 1b 01 00 00
40: 00 00 00 00 00 00 00 00 01 ac 13 48 08 20 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 10 00 42 00
b0: 02 80 00 00 10 2c 00 00 12 cc 64 00 42 00 12 d0
c0: 00 00 00 00 00 00 40 00 18 00 01 00 00 00 00 00
d0: 1f 08 08 00 00 04 00 00 06 00 00 80 02 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
    Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 27
    Region 0: Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
    Region 4: Memory at 600004000 (64-bit, non-prefetchable) [size=256]
    Capabilities: [80] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [b0] MSI-X: Enable+ Count=17 Masked-
        Vector table: BAR=0 offset=00002000
        PBA: BAR=4 offset=00000000
    Capabilities: [c0] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
        DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
    Capabilities: [1b8 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [300 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [900 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
              PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=48128ns
        L1SubCtl2: T_PwrOn=10us
    Kernel driver in use: nvme
00: b7 15 09 50 06 04 10 00 01 02 08 01 00 00 00 00
10: 04 00 00 c0 00 00 00 00 00 00 00 00 00 00 00 00
20: 04 40 00 c0 00 00 00 00 00 00 00 00 b7 15 09 50
30: 00 00 00 00 80 00 00 00 00 00 00 00 1b 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 01 90 03 00 08 00 00 00 00 00 00 00 00 00 00 00
90: 05 b0 8a 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 c0 10 80 00 20 00 00 04 00 00 00 00 00 00 00
c0: 10 00 02 00 02 8f 00 10 1f 28 00 00 43 a8 45 00
d0: 42 01 12 10 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 12 08 10 00 00 04 00 00 0e 00 00 00
f0: 03 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00

pi@raspberrypi:~ $ cat /sys/class/nvme/nvme0/model && cat /sys/class/nvme/nvme0/firmware_rev
SanDisk Ultra 3D NVMe                   
211210WD

Moving the NVMe to Pi5, power up -> SUCCESS!

pi@m2hattest:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
pi@m2hattest:~ $ cat /sys/class/nvme/nvme0/model && cat /sys/class/nvme/nvme0/firmware_rev
SanDisk Ultra 3D NVMe                   
211210WD
pi@m2hattest:~ $ sudo nano /boot/firmware/config.txt
pi@m2hattest:~ $ lspci -tvnn
-+-[0000:00]---00.0-[01]----00.0  Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009]
 \-[0001:00]---00.0-[01]----00.0  Device [1de4:0001]
pi@m2hattest:~ $ sudo nvme --smart-log /dev/nvme0n1 -H
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning            : 0
      Available Spare[0]             : 0
      Temp. Threshold[1]             : 0
      NVM subsystem Reliability[2]   : 0
      Read-only[3]                   : 0
      Volatile mem. backup failed[4] : 0
      Persistent Mem. RO[5]          : 0
temperature             : 35°C (308 Kelvin)
available_spare             : 100%
available_spare_threshold       : 10%
percentage_used             : 0%
endurance group critical warning summary: 0
Data Units Read             : 938,560 (480.54 GB)
Data Units Written          : 857,454 (439.02 GB)
host_read_commands          : 11,392,058
host_write_commands         : 5,927,484
controller_busy_time            : 72
power_cycles                : 559
power_on_hours              : 230
unsafe_shutdowns            : 146
media_errors                : 0
num_err_log_entries         : 1
Warning Temperature Time        : 0
Critical Composite Temperature Time : 0
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0
pi@m2hattest:~ $ sudo lspci -vvv | grep -E 'nvme|LnkSta'
        LnkSta: Speed 5GT/s, Width x1
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
        LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
    Kernel driver in use: nvme
        LnkSta: Speed 5GT/s, Width x4
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
        LnkSta: Speed 5GT/s, Width x4
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-