search

geerlingguy / raspberry-pi-pcie-devices

Raspberry Pi PCI Express device compatibility database
http://pipci.jeffgeerling.com
GNU General Public License v3.0
1.6k stars 145 forks source link

Test various M.2-key NVMe SSD drives #48

Open geerlingguy opened 3 years ago

geerlingguy commented 3 years ago

I have on hand and would like to test the following with the Pi via an M.2 adapter, to come around full circle on the video I posted in Summer 2020: https://youtu.be/3yTyhR0Adao:

I already tested the 970 EVO Plus and the XPG SX6000 for the Raspberry Pi Compute Module 4 Review, but I'd like to go more in depth, and compare all the cards in terms of whether they work direct through a PCIe adapter, and how fast they are using the disk-benchmark.sh script.

geerlingguy commented 3 years ago

Besides general performance, I'd also like to discuss booting off an NVMe drive—specifically, since direct booting from one is not supported, how do you get the Pi to initialize off a microSD card then finish the boot off an NVMe?

Magnum-Pl commented 3 years ago

I tested a 980 Pro and wasn't able to get it to work. I tried a 970 EVO first and was able to see that using lspci and lsblk. The Pi won't even boot with the 980 attached. I can plug it in after the Pi boots, but I can't see it with lspci or lsblk.

I used a Cablecc PCI-E 3.0 x4 M.2 NGFF M-Key NVME adapter

geerlingguy commented 3 years ago

@Magnum-Pl thanks for sharing. Weird that the 980 would behave any different. Did you confirm it works in another computer or USB adapter? Also, what capacity is it? I've only been testing with 512GB or less, if that makes any difference.

Magnum-Pl commented 3 years ago

They're both 500GB. The 980 works fine with my Mac and PC. I've been using this Wanfocyu USB-C NVMe adapter. They're both formatted NTFS. I tried FAT32 on the 980, but that didn't work either. The adapter doesn't work when plugged into the USB 2 ports on the IO board, but I have a Syba SD-PEX20199 on the way.

garlick commented 3 years ago

I can verify that the Samsung 970 EVO (without the "plus") also works great.

I'm using this Simtech adapter.

ThomasKaiser commented 3 years ago

how fast they are using the disk-benchmark.sh script.

Just a quick note: Based on a lot of testing years ago with RK3399 boards and NVMe SSDs I came to the conclusion that different settings are needed to test fast SSDs compared to rather slow SD cards.

hdparm is a problem in itself since it uses a hardcoded block size of 128K which was fine when this was defined (last century) but doesn't represent today's access patterns any more. Simply compare hdparm numbers with the results from iozone -e -I -a -s 100M -r 4k -r 128k -r 16384k -i 0 -I 1.

And when using iozone to test for sequential transfer rates I found that 100M with iozone is not enough with SSDs which can be easily observed comparing a 100M run with 1000M.

geerlingguy commented 3 years ago

@ThomasKaiser - Just noting here (also noted in another issue) that I updated the disk-benchmark.sh script quite a bit, to use much larger sequential size for the testing, and a different set of tests all based on fio and iozone (discarding dd/hdparm).

See: https://github.com/geerlingguy/raspberry-pi-dramble/blob/master/setup/benchmarks/disk-benchmark.sh

Also, I've been doing a bunch of testing with a WD_Black SN750 500 GB drive in the MirkoPC, and one interesting thing to note is it consumes a good bit of power at full tilt. I wish the MirkoPC accepted more than 5v/3A power input...

t5b6de commented 3 years ago

I wanna test a nvme samsung 950 pro 250gb parallel with a vl805 4 port usb card connected to a 2 port pcie switch.

if someone is interested into that results. i may reply to that posts here. Maybe about end of the week i should have some first results

Johnnie390 commented 3 years ago

Trying to test a Samsung 980 Pro* as NVME boot device on a CM4 Lite. At boot - "cannot open nvme", EEPROM is up to date (19.05.21). A Synology SNV3400-400G NVME SSD boots as expected using the abovementioned EEPROM.

t5b6de commented 3 years ago

You to not have any other pcie switch or else inbetween?

to my tests:

With the pcie switch plugged in, i cannot boot from the nvme nor the usb stick connected to the card. If i plug in the card directly into that slot, the rpi firmware can boot of it without any problems.

My pcie to m.2 adapter does not fit into that x1 slot on the pi board, its a x4 pcb. So i cannot test it directly. Booting is from sd card, which modified cmdline.txt which points to /dev/nvme0n1p1 where / is located.

Performance with a real anvme (even over a pcie switch) is awesome. never thought that a raspberry pi can run that smoothly.

about 420MiB/s are the limit through the PCIe switch. And all does work out of the box, i do not have to change anything regarding to the kernel itself or device tree.

Kind regards.

Johnnie390 commented 3 years ago

I am using an Oratek Tofu carrier board to test the Samsung 980 Pro. No switches anywhere. As already stated, the Synology SNV3400-400G NVME SSD will boot. I am not sure whether it is the CM4 Lite EEPROM (still somewhat unstable for NVME boot / work-in-progress) or the firmware on the Samsung 980 Pro. I was hoping the Samsung would work seeing as it is expensive and using the Elpis controller. There are still so many variables and unknowns in this equation.

geerlingguy commented 3 years ago

@Johnnie390 - FYI the 980 Pro is a bit newer and maybe there's some limitation due to that? I've only tested up to the 970 (which did work).

Johnnie390 commented 3 years ago

I am aware that the device is relatively new. CM4 Lite EEPROM NVME booting is also "relatively new" and somewhat flaky. As stated above, it could be an EEPROM problem and/or a 980 Pro firmware/millicode glitch. I would tend towards the EEPROM though. The Samsung devices have always been "well behaved" (for me, at least...).

adminy commented 3 years ago

I have the 1TB version of XPG.

here is the log I get in relation to pcie

pi@raspberrypi:~ $ sudo dmesg | grep -i pcie
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=DC:A6:32:FF:4E:43 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=PARTUUID=a54b457f-02 rootfstype=ext4 fsck.repair=yes rootwait nvme_core.default_ps_max_latency_us=0 pcie_aspm=off
[    1.151993] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.152043] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.152144] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x07ffffffff -> 0x0000000000
[    1.152254] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[    1.744723] brcm-pcie fd500000.pcie: link down
[    2.176233]     pcie_aspm=off

The ssd is not recognised. I also tried increasing the BAR and its not helping.

adminy commented 3 years ago

My SSD is an XPG M.2 2280 SX8200PNP 1TB. I switched to ubuntu for change of perspective and I get this:

ubuntu@ubuntu:~$ sudo dmesg | grep pcie
[    0.997358] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    0.997393] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    0.997468] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    0.997540] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[    1.047135] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[    1.047488] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.056225] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    1.056425] pcieport 0000:00:00.0: PME: Signaling with IRQ 50
[    1.056842] pcieport 0000:00:00.0: AER: enabled with IRQ 50

ubuntu@ubuntu:~$ sudo dmesg | grep nvme
[    1.505589] nvme nvme0: pci function 0000:01:00.0
[    1.505666] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[  122.009439] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
[  122.016559] nvme nvme0: Removing after probe failure status: -19

ubuntu@ubuntu:~$ sudo lspci -vvv
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) (prog-if 00 [Normal decode])
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 50
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 00000000-00000fff [size=4K]
    Memory behind bridge: c0000000-c00fffff [size=1M]
    Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: [48] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
        DevCap: MaxPayload 512 bytes, PhantFunc 0
            ExtTag- RBE+
        DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
            ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s (ok), Width x1 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
        RootCap: CRSVisible+
        RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
        RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
             AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
             AtomicOpsCtl: ReqEn- EgressBlck-
        LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported, DRS-
             DownstreamComp: Link Down - Not Determined
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
        RootCmd: CERptEn+ NFERptEn+ FERptEn+
        RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
             FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
        ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
    Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
    Capabilities: [240 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=10us LTR1.2_Threshold=25600ns
        L1SubCtl2: T_PwrOn=10us
    Kernel driver in use: pcieport

01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03) (prog-if 02 [NVM Express])
    Subsystem: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 50
    Region 0: Memory at 600000000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
        Vector table: BAR=0 offset=00002000
        PBA: BAR=0 offset=00002100
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [158 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [178 v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [180 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=25600ns
        L1SubCtl2: T_PwrOn=10us

Probe failure -19 suggests there is something wrong with the device but that's not the case as its working fine plugged in through an adapter to the mac.

BeauSlim commented 3 years ago

I have a Waveshare CM4-IO-BASE-A and have tried the ADATA XPG SX8200PNP 256GB and 512GB SSDs in it. Performance seems decent, but I am getting a weird CPU throttling error when running Jeff's benchmark script that I have described on the Pi forums

I haven't tried booting from nvme yet.

dbeckwith01 commented 2 years ago

Has any one tried a WD-Black SN770 with the RPI CM4? I tired on today and I couldn't get it to see the drive. I have tested SN750 which worked fine. I don't have logs I will post them when I do.

Update I tired this again on a CM4IO board and the SN770 isn't seen on boot. I tried also updating the bootloader on the CM4

dmesg | grep pci
[    1.229964] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.230014] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.230117] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.230227] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
[    1.836201] brcm-pcie fd500000.pcie: link down
echaz commented 2 years ago

Has anyone tried larger capacity drives? I got this from the Sabrent 2TB 2242 drive

[    1.342103] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.342140] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.342233] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.342331] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[    1.942393] brcm-pcie fd500000.pcie: link down
aurel32 commented 2 years ago

I just tried a WD Blue SN570 2TB, and it's not detected, I get the same link down message as above.

eavanvalkenburg commented 2 years ago

Has any one tried a WD-Black SN770 with the RPI CM4? I tired on today and I couldn't get it to see the drive. I have tested SN750 which worked fine. I don't have logs I will post them when I do.

Update I tired this again on a CM4IO board and the SN770 isn't seen on boot. I tried also updating the bootloader on the CM4

dmesg | grep pci
[    1.229964] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.230014] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.230117] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.230227] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
[    1.836201] brcm-pcie fd500000.pcie: link down

Have you ever been able to get this going?

dbeckwith01 commented 2 years ago

Hi @eavanvalkenburg

No I haven't gotten anywhere with the SN770 drives. Sorry.

aurel32 commented 2 years ago

I haven't been able to get the WD Blue SN570 to work, so I bought another SBC, which does not have this limitation.

yuqo2450 commented 1 year ago

Hi there, any recommendations for a 2230 or 2242 NVMe, that will work with the Waveshare CM4-IO-BASE-B board? Would be amazing, if someone has tested so far.

dmeziere commented 1 year ago

I tried without success a WD Blue SN580 2TB on this adapter without success on CM4 + CM4IO. It does not appear in an lspci or lsblk. I also played arround with PCIe ranges in the dtb. Detected fine in my PC. If anyone has any clue, i am interested. If anyone knows a 2TB NVME that works, i am also.

[update] I didn't play enough with the BAR range. It seems to work after all !