geerlingguy
commented
3 years ago Are you using an external PCIe riser or switch, or just plugging into the official IO Board directly?
Closed pastudan closed 3 years ago
geerlingguy
commented
3 years ago Are you using an external PCIe riser or switch, or just plugging into the official IO Board directly?
pastudan
commented
3 years ago Plugging in directly
geerlingguy
commented
3 years ago I split out the two cards since even though they use the same family chipset, there could be differences. I'm also going to add my pictures to the first comment on this issue :)
pastudan
commented
3 years ago Awesome, thanks Jeff!!
On Tue, Feb 2, 2021 at 12:47 PM Jeff Geerling notifications@github.com wrote:
I split out the two cards since even though they use the same family chipset, there could be differences. I'm also going to add my pictures to the first comment on this issue :)
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/64#issuecomment-771972395, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ4OISNDXYU7FVFK3BFIV3S5BQGVANCNFSM4W3Y4VDQ .
ThomasKaiser
commented
3 years ago @PastuDan according to #85 JMB585 is confirmed to work. Maybe that helps with your JMB582 as well?
geerlingguy
commented
3 years ago Just dropping some notes in here:
# Following my cross-compile instructions:
# https://github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile
make menuconfig
> Device Drivers
> Serial ATA and Parallel ATA drivers (libata) (enable this - M)
> AHCI SATA support (enable this - M)
> Marvell SATA support (enable this - M)Just wiring it up with the new kernel in place:
01:00.0 SATA controller: JMicron Technology Corp. Device 0585 (prog-if 01 [AHCI 1.0])
Subsystem: JMicron Technology Corp. Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 48
Region 0: I/O ports at <unassigned> [disabled]
Region 1: I/O ports at <unassigned> [disabled]
Region 2: I/O ports at <unassigned> [disabled]
Region 3: I/O ports at <unassigned> [disabled]
Region 4: I/O ports at <unassigned> [disabled]
Region 5: Memory at 600010000 (32-bit, non-prefetchable) [size=8K]
[virtual] Expansion ROM at 600000000 [disabled] [size=64K]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/8 Maskable- 64bit+
Address: 00000000fffffffc Data: 6540
Capabilities: [c0] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM not supported, Exit Latency L0s <256ns, L1 <8us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [160 v1] Power Budgeting <?>
Capabilities: [1b8 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [300 v1] #19
Capabilities: [900 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
L1SubCtl2:
Kernel driver in use: ahci
Kernel modules: ahciAnd in dmesg:
[ 1.208280] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 1.208321] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 1.208406] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 1.208497] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
[ 1.243054] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[ 1.243451] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[ 1.243483] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.243511] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 1.243609] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 1.243856] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 1.247571] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[ 1.247838] pci 0000:01:00.0: [197b:0585] type 00 class 0x010601
[ 1.247906] pci 0000:01:00.0: reg 0x10: [io 0x0000-0x007f]
[ 1.247947] pci 0000:01:00.0: reg 0x14: [io 0x0000-0x007f]
[ 1.247987] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x007f]
[ 1.248026] pci 0000:01:00.0: reg 0x1c: [io 0x0000-0x007f]
[ 1.248066] pci 0000:01:00.0: reg 0x20: [io 0x0000-0x007f]
[ 1.248106] pci 0000:01:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 1.248147] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 1.248405] pci 0000:01:00.0: PME# supported from D3hot
[ 1.248501] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[ 1.251989] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 1.252043] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[ 1.252077] pci 0000:01:00.0: BAR 6: assigned [mem 0x600000000-0x60000ffff pref]
[ 1.252107] pci 0000:01:00.0: BAR 5: assigned [mem 0x600010000-0x600011fff]
[ 1.252139] pci 0000:01:00.0: BAR 0: no space for [io size 0x0080]
[ 1.252163] pci 0000:01:00.0: BAR 0: failed to assign [io size 0x0080]
[ 1.252187] pci 0000:01:00.0: BAR 1: no space for [io size 0x0080]
[ 1.252210] pci 0000:01:00.0: BAR 1: failed to assign [io size 0x0080]
[ 1.252234] pci 0000:01:00.0: BAR 2: no space for [io size 0x0080]
[ 1.252257] pci 0000:01:00.0: BAR 2: failed to assign [io size 0x0080]
[ 1.252281] pci 0000:01:00.0: BAR 3: no space for [io size 0x0080]
[ 1.252304] pci 0000:01:00.0: BAR 3: failed to assign [io size 0x0080]
[ 1.252328] pci 0000:01:00.0: BAR 4: no space for [io size 0x0080]
[ 1.252351] pci 0000:01:00.0: BAR 4: failed to assign [io size 0x0080]
[ 1.252377] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 1.252409] pci 0000:00:00.0: bridge window [mem 0x600000000-0x6000fffff]
[ 4.281691] pci 0000:00:00.0: enabling device (0000 -> 0002)
...
[ 4.043411] libata version 3.00 loaded.
[ 4.384453] ata1: SATA max UDMA/133 abar m8192@0x600010000 port 0x600010100 irq 48
[ 4.384471] ata2: SATA max UDMA/133 abar m8192@0x600010000 port 0x600010180 irq 48
[ 4.384485] ata3: SATA max UDMA/133 abar m8192@0x600010000 port 0x600010200 irq 48
[ 4.384498] ata4: SATA max UDMA/133 abar m8192@0x600010000 port 0x600010280 irq 48
[ 4.384511] ata5: SATA max UDMA/133 abar m8192@0x600010000 port 0x600010300 irq 48
[ 4.700649] ata1: SATA link down (SStatus 0 SControl 300)
[ 5.013874] ata2: SATA link down (SStatus 0 SControl 300)
[ 5.328680] ata3: SATA link down (SStatus 0 SControl 300)
[ 5.647804] ata4: SATA link down (SStatus 0 SControl 300)
[ 5.963220] ata5: SATA link down (SStatus 0 SControl 300)(I don't have any devices plugged in currently.) No lockup, yay!
geerlingguy
commented
3 years ago $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 7.3T 0 disk
sdb 8:16 0 7.3T 0 disk
sdc 8:32 0 7.3T 0 disk
sdd 8:48 0 7.3T 0 disk
mmcblk0 179:0 0 59.5G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 59.2G 0 part /(Even working through my PCI switch, yay!)
geerlingguy
commented
3 years ago I'm seeing around 395 MB/sec (3.16 Gbps) through the card during a RAID 5 parity calculation (initializing the disk through OMV, which seems to be using mdadm).
Additionally, it seems like one of the four drives I was given is... dead on arrival :( — beggars can't be choosers, but what sad luck on my first try with 'real' NAS gear?
geerlingguy
commented
3 years ago | Benchmark | Result 1 | Result 2 | Result 3 | Average |
|---|---|---|---|---|
| hdparm | 274.72 MB/sec | 307.11 MB/sec | 311.27 MB/sec | 297.70 MB/sec |
| dd write | 98.20 MB/sec | 105.00 MB/sec | 104.00 MB/sec | 102.40 MB/sec |
| 4k read | 19.04 MB/sec | 19.44 MB/sec | 19.37 MB/sec | 19.28 MB/sec |
| 4k write | 2.83 MB/sec | 2.90 MB/sec | 2.98 MB/sec | 2.90 MB/sec |
Note that the above numbers are not indicative of the maximum performance of this card, but rather the entire system with RAID 5. I'm going to run the same tests on a RAID 0 volume later, after I do some more network testing, to see how the numbers differ. And all these numbers are also on spinning disks too (in case you're wondering why 4K write is soooo slow).
ThomasKaiser
commented
3 years ago What is this for a setup? RAID-5 set up by mdadm? If so which settings and which devices?
And I'm still curious which purpose sequential read and write tests should have that are done with 128K or even 8K block size...
geerlingguy
commented
3 years ago @ThomasKaiser - It's mostly a way for me to have a line of comparison back to my original tests starting in around 2014, ranging from slow microSD cards up to SSDs. Note that there is no single benchmark I've found that can really be well-rounded that can target every kind of storage device... I just like the consistency.
I will likely also be doing some fio comparisons, and in these GitHub issues I mostly document things for my own reference later.
The specific benchmark script I'm using here is https://github.com/geerlingguy/raspberry-pi-dramble/blob/master/setup/benchmarks/disk-benchmark.sh
(Also see the latest edits I made to the comment above yours for more on the setup in this particular instance.)
geerlingguy
commented
3 years ago Some NFS benchmarks on the same RAID 5 setup (mdadm via OMV), over the 2.5 Gbps network (I am able to get 2.2 Gbps from Pi to Mac, ~0.8 Gbps from Mac to Pi. Not sure why it's not very asymmetric.).
$ time dd if=/dev/zero of=/Volumes/NFS/test bs=1024k count=8k
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 376.259394 secs (22829821 bytes/sec)
real 6:17.02
user 0.023
sys 52.107(22.83 MB/sec... not super amazing :/)
And atop is showing all three disks hitting 85%+ utilization, with md0 hitting 100% quite frequently.
geerlingguy
commented
3 years ago And using pv over the 2.5 GbE interface (confirmed again I can get ~800 Mbps from Mac to Pi, 2.2 Gbps from Pi to Mac, just to be certain):
# From Pi to Mac, 8 GiB file:
$ pv /Volumes/NFS/test > ~/Downloads/test
8.00GiB 0:01:08 [ 119MiB/s] [===============================================================================>] 100%
# From Mac to Pi, same file.
$ pv ~/Downloads/test > /Volumes/NFS/test
8.00GiB 0:07:04 [19.3MiB/s] [===============================================================================>] 100% And pv over the 1 GbE built-in interface:
# From Mac to Pi, 8 GiB file.
$ pv /Volumes/NFS/test > ~/Downloads/test
8.00GiB 0:01:31 [89.2MiB/s] [===============================================================================>] 100%
# From Pi to Mac, same file:
$ pv ~/Downloads/test > /Volumes/NFS/test
8.00GiB 0:24:45 [5.51MiB/s] [===============================================================================>] 100%Going to test out RAID 0 next, just to see what the max performance is for SATA + mdadm raid. These numbers are... not overwhelming. But hardware RAID may be able to fix it. We'll see.
geerlingguy
commented
3 years ago Using RAID 0 on the 2.5G interface:
# Pi to Mac:
$ pv /Volumes/NFS/test > ~/Downloads/test
8.00GiB 0:01:06 [ 123MiB/s] [===============================================================================>] 100%
# Mac to Pi:
$ pv ~/Downloads/test > /Volumes/NFS/test
8.00GiB 0:05:38 [24.2MiB/s] [===============================================================================>] 100% And just a single drive (sda) on the 2.5G interface:
# Pi to Mac:
$ pv /Volumes/NFS/test > ~/Downloads/test
8.00GiB 0:01:03 [ 128MiB/s] [===============================================================================>] 100%
# Mac to Pi:
$ pv ~/Downloads/test > /Volumes/NFS/test
8.00GiB 0:05:36 [24.4MiB/s] [===============================================================================>] 100% I'm going to test the same thing, taking the PCIe switch and 2.5GbE card out of the loop, to see if that's causing any kind of bottleneck with the writes.
Edit: Nope, still crazy slow.
$ pv ~/Downloads/test > /Volumes/NFS/test
500MiB 0:01:35 [4.39MiB/s] [===> ] 6% ETA 0:24:18Where is the bottleneck???
geerlingguy
commented
3 years ago And to be clear, testing the drive itself:
hdparm - 252.83 MB/sec
dd - 127 MB/s
4k rand read - 30.16 MiB/sec
4k rand write - 4.67 MiB/secAlso testing a few copies of the 8 GiB file between SDA and the microSD card — I see 36 MB/sec written to the card, and 41 MB/sec written back from the card to SDA (the literal upper limit of how fast bits can go to/from the microSD card. The 8 GiB file seems to have filled up the RAM file cache, which is helpful for getting actual speeds and not cached speeds.
I even copied a file between two drives SDA/SDB, and got 173 MiB/sec (346 MiB/sec avg with both drives taken into account, saw peaks around 395 MiB/sec with atop):
pi@raspberrypi:/srv/dev-disk-by-uuid-935fd429-d20a-469a-a9a4-4a34aeac8ce7/NFS $ pv ./test > /srv/dev-disk-by-uuid-bb10ee88-4f06-4500-bf0a-8a4e079eba70/NFS/test
8.00GiB 0:00:47 [ 173MiB/s] [========================================================================>] 100% So I'm not sure where the slow speed on the NFS copies is coming from. (I even tried with/without async on the NFS export options.)
geerlingguy
commented
3 years ago I also enabled SMB/CIFS and tried copying through there... same issue. Writes over the network are incredibly slow in comparison to reads (especially on the 1 Gbps interface), and I can't see anything that would seem to be a culprit.
$ uname -a
Linux raspberrypi 5.10.20-v8+ #1 SMP PREEMPT Fri Mar 5 15:27:59 UTC 2021 aarch64 GNU/LinuxJust for reference.
Also tried rsync directly to the drive, and even FTP! All of them averaging around 5 MiB/sec writing to the Pi on the 1 Gbps network, and 20-25 MiB/sec writing to the Pi on the 2.5 Gbps network. Soooo strange.
geerlingguy
commented
3 years ago Added an overclock to /boot/config.txt:
over_voltage=6
arm_freq=2147
gpu_freq=750Rebooted, tried again, still just as slow (maybe a tiny bit faster? Not easy to measure though without a very very long burn-in test.)
Also tried scp and rsync to the home directory on microSD, still only seeing 4-5 MB/sec tops over 1 Gbps ethernet. Weird. Does OMV cause some strange overhead on the Pi's networking stack (I installed on top of Pi OS, I didn't use the pre-flashed image. I do know it does 'some' sort of networking configuration when it installs... I wonder if it blows away something that is necessary to keep fast writes to the Pi.
I might have to completely re-test everything without OMV just to see if I can get NFS much faster.
ThomasKaiser
commented
3 years ago It's mostly a way for me to have a line of comparison back to my original tests starting in around 2014
Well, it doesn't make a lot of sense since back then in 2014 all you had to test with were rather crippled Raspberry Pis when speaking of I/O capabilities (both network and disk storage behind a single Hi-Speed USB lane). And most probably your choice of tools back then was influenced by the RPi community.
Anyway: this whole test methodology makes no sense at all and the same is true for using anachronistic RAID today in such a setup. You staple layers of complexity on another without caring about the involved drawbacks. For example if you don't aline RAID parameters with disk parameters you end up with horribly low write performance since every block access at the filesystem layer results in accessing 2 blocks at the block device layer below.
You also need at least 3 machines to do network testing (the 3rd always as a reference to identify bottlenecks, in your above testings most probably your Mac being the culprit[1]). If networking is not confirmed to be free of problems (iperf +940 Mbits/sec in both directions without retransmits) then it makes no sense to continue with NAS tests since one known bottleneck is already in the mix that will invalidate all further results.
While I understand that this whole project is about the joy of tinkering and 'just for fun' it's a bit sad to use inappropriate methodology since the time and efforts you and others spend on generating 'performance numbers' are somewhat wasted.
If the goal is to determine realistic real-world maximum sequential throughput rates for a storage device then you clearly can neither use hdparm (limited by a horribly low block size of hardcoded 128K and by a test execution of just 3 seconds) nor dd with even worse parameters (8K block size and 50k iterations). You need to use at least 1 MB block sizes since this is what happens in real-world situations as well since many years and a test execution large enough to accommodate for ramping up cpufreq or leaving (deep) sleep states inside SSD storage. So we're back at this again :)
[1] As an example for why 'passive benchmarking' (in contrast to active benchmarking) between a Mac and a Linux system can show horribly low write performance through the network 'nobody can explain' look back almost 1.5 decades ago when we needed to adjust the net.inet.tcp.delayed_ack parameter in MacOS to cope with more modern TCP/IP stack implementations on another OS at the other end of the network cable.
ThomasKaiser
commented
3 years ago Does OMV cause some strange overhead on the Pi's networking stack
No (I should know since having provided dedicated OMV images for many SBC RPi included for some time and most of the installation routine's code by me).
The OMV installation routine does change something wrt networking when executed on Armbian installs but this doesn't apply here (Raspbian or 'Pi OS' as it's called today). Also on Armbian installs there's a lot of tuning here and there but this also does not apply here on the RPi (one thing I would check is /proc/interrupts prior and after benchmark execution to see whether cpu0 makes up for a nice bottleneck wrt IRQ processing).
Wrt test methodology: you need to test from bottom to top (storage individually, network individually, only then both combined – and all of this with appropriate parameters). As repeated frequently: testing with inappropriate block sizes will just provide numbers without meaning (which use case is your dd test with bs=8k count=50k representing?).
Please keep in mind that hdparm's benchmark modes have been developed last century. Back then DMA was a hot thing compared to PIO and 128K block sizes could be considered huge. Back then HDDs had capacities no average operating system update we now download through the Internet every other day would fit on. Things have changed, use cases have changed, testing methodology needs to be changed accordingly.
geerlingguy
commented
3 years ago @ThomasKaiser - Do you have a benchmarking script or suite that would be more appropriate for disk-based benchmarking? I built the original script with hdparm/dd/iozone because that was the best combo to try to test horrendously-slow and behaving-like-old-IDE-drives microSD cards back in the day, and the tests still gave useful (if incomplete) numbers for hard drives over the old USB 2.0 busses.
But I'm happy and willing to update my scripts to match the modern performance/standards on the Pi. Is it a matter of just switching to fio + iozone for 1M block size sequential tests (with 8+ GB write so caches aren't involved—at least on the Pi) plus 4k random IO? (I feel like the 4k rand test is the best proxy for worst case performance, still, no matter what kind of drive).
I really wish there were a standard widely-used disk test script like there are for CPU/memory/GPU quite frequently, and maybe there is, but it seems like most people just have their custom fio or iozone setup and many people don't even share the parameters :(
My benchmarks may be inadequate, but by golly, at least I share the script in the open so people can replicate them :P
geerlingguy
commented
3 years ago @ThomasKaiser - What do you think of these changes?
https://github.com/geerlingguy/raspberry-pi-dramble/commit/a0b1708a92a9abc9b13343735d3b34beb67af626
It runs a 1M sequential read test using fio. And I wanted to use fio to test 1M sequential writes... but that destroys a mounted disk (I want to be able to run the benchmarks on an already-existing/in-use disk, it looks like fio's write tests are destructive and meant only for benchmarking / testing on a test environment, not running against production situations).
So instead I also run 1M sequential tests using iozone, as well as 4k random tests using iozone.
First set of results seems good from this script: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/92#issuecomment-792883731 — however I find it odd that iozone's performance numbers don't quite match up to fio's (though the sequential write speeds are close between the two).
geerlingguy
commented
3 years ago All right, testing speeds on an Asustor NAS too, I'm seeing the same really weird discrepancy between Mac-to-device vs device-to-Mac.
I swapped out the Cat5e patch cable for Cat8 and a Cat7. No difference (Pi to Mac always 2.2 Gbps, NAS to Mac always 2.35 Gbps... Mac to Pi and Mac to NAS fluctuating wildly around 500-700 Mbps).
I also swapped out Thunderbolt 3 cables (tried three different cables). No difference, very close results.
Tried my gigabit port on my CalDigit Thunderbolt 3 dock, and it's solidly hitting 940 Mbps from Mac to Pi and Mac to NAS. So something is really funky with the OWC Thunderbolt 3 10 Gbps adapter. Used the same port on the Mikrotik switch too, and even tried a different port and copper SFP+ adapter, and nothing made a difference.
I'm going to turn off the OWC adapter and let it cool down and see if that makes any difference. Would stink to have to find another $100+ 10 gig adapter for the Mac. Any good USB 2.5 Gbps adapters out there?
Edit: Just ordered an Inateck 2.5G Ethernet Adapter ET1001 USB-C adapter. Better to have more than one option.
(And if it's heat-related the CalDigit Connect 10G wouldn't be of any help either.)
Edit: It's not heat-related. 10G adapter cooled down to ambient temp after an hour... still showing inconsistent results (836/597/588/562/837/584/839/816/573/378 averaged to 660 Mbps). Still perfectly clean the other way (device to Mac) though. Grr!
pastudan
commented
3 years ago Jeff, I can’t stress enough how much I appreciate your thorough testing!! This is immensely helpful while I design around these chips :)
geerlingguy
commented
3 years ago @PastuDan - Glad to help! At least I can determine now that I don't think there's any issue with the JMB582/585—they have performed flawlessly, even through the PCIe switch. I really need to figure out this networking issue on my Mac though!
geerlingguy
commented
3 years ago Doing more debugging—I've even changed out the cable to the NAS (which I already did once, but once more with a brand new and known-good-to-10G-cable), and that didn't help. I also confirmed both ends of the connection are showing as 'Full' duplex. The NAS at 2500Mb/s (ethtool eth0), and my Mac at 10 Gbps (in Network preference pane).
Also confirmed that all devices are configured with normal MTU (1500).
I also monitored the traffic with Wireshark, and besides a few TCP retransmissions (not a huge percentage), there was nothing suspect. I even uninstalled Little Snitch on my Mac entirely, but that didn't make a difference either.
Might also try on my wife's 2016 MacBook Pro and see if it has the same issues, to see if it's something software-related on my Mac.
geerlingguy
commented
3 years ago I also tested traffic between my router and the Asustor, and it's consistently 940 Mbps both ways. So something's definitely up with my Mac. Grr.
geerlingguy
commented
3 years ago All right, so more notes as I get back to testing... soon:
I have a 2018 MacBook Pro running macOS 10.15 Catalina. This device works at 1gbe without any config, but the driver from the Realtek website must be installed for 2.5gbe operation. Installation is very quick and simple and no more config is required after rebooting. One thing of note is that Apple informs you that the Realtek driver will not be compatible with the next version of macOS (10.16), so I'll be curious to see if Realtek releases an update for the next release. I hope so.
We'll see. Worst case, I do all the rest of my testing over on my HP desktop, with the ASUS 10G adapter in it.
mi-hol
commented
3 years ago do all the rest of my testing over on my HP desktop, with the ASUS 10G adapter in it.
I'd go that route as I had bad experience to get networking on macOS 10.13 and later to work at higher speeds in a huge DC environment too. macOS is targeted at consumers, getting proper technical support for complex networking environments turned out to be impossible (at least in Germany).
geerlingguy
commented
3 years ago The plot thickens. I was using a MikroTik 10G Switch/Router (in switch mode) for all this testing, but once I started seeing the HP desktop also have weird issues, using a Mellanox ConnectX2 card, and the ASUS adapter, where it would be 2.3 Gbps down but only 600-800 Mbps up... I'm doubting everything I know.
I have no idea why bandwidth is behaving weird on the MikroTik, but since I now isolated every other potential fault, I decided to bite the bullet an buy a full-fledged QNAP switch (this one: QNAP QSW-M2108-2C 10-port/8x 2.5 GbE), and re-run all the tests on it.
And lo and behold... 2.3 Gbps both directions, all the time. So, it seems like it may not have had anything to do with the Mac. And now I'm scratching my head over how the MikroTik could be having issues. I have it booted into SwOS (so the router overhead is not there), it's not super hot, and it always gives 2.3 Gbps in one direction.
Until I get back my RMA'd OWC adapter, I won't be able to test beyond 2.5 Gbps.
ANYWAYS, that's a HUGE tangent on this card—and an expensive one—so I'll be getting back to my performance testing soon :)
mister2d
commented
3 years ago Do you have a running list of known good hardware yet? I wouldn't mind testing this out myself.
Thanks.
geerlingguy
commented
3 years ago @mister2d - My blog and YouTube channel will have a post in just a bit with all the details of my 2.5G Pi NAS build (the hardware side), so check that out when it's released!
Otherwise, all the info in this repository gets aggregated into https://pipci.jeffgeerling.com (I need to run through and update some things though... haven't taken a pass for a couple weeks as I've been heads-down in some projects).
pastudan
commented
3 years ago @geerlingguy following your instructions I was able to get the JMB585 chipset card working, but not the JMB582. Still having issues with the kernel panic at boot.
Unfortunately my custom board is based around the JMB582, and switching to the 585 would be rather costly if you don’t need 5 drives. At volume pricing they run $13 and $25 respectively.
geerlingguy
commented
3 years ago Video is up: https://www.youtube.com/watch?v=vBccak8f-VY
And @PastuDan, I'll try testing out the 582 you sent soon!
geerlingguy
commented
3 years ago All right, finally had some time to plug the thing in... on a fresh boot with just the plain old Pi OS ARM64 build installed, I also get the kernel panic if I have the card plugged in. I'm going to pull out my USB/UART adapter and get a dump of the actual text for comparison.
geerlingguy
commented
3 years ago Unfortunately had trouble getting the output (maybe the kernel panic happens too early to start output over uart?), so here's a picture:
pastudan
commented
3 years ago Bummer. Glad to know I'm not crazy though! I didn't think to tap the UART for those kernel logs, that's a good idea.
I'll see if I can give that a try later. There's got to be some helpful output at the top of those logs that my video doesn't capture.
geerlingguy
commented
3 years ago The panic might occur too early for the console to start up, in which case I wonder if the settings under Enabling early console (earlycon) for Linux in the UART configuration guide might be required. Unfortunately I need to drop back to another project right now or else I'd give it a go.
geerlingguy
commented
3 years ago All right, I'm going to call this as 'won't work', at least for the JMB582. Weird that other related chips in the same family do, but for now I'll update the site with the latest findings so others can just move on to a SATA controller that is known working (since there are a number now that have been tested and verified!).
solstyce
commented
3 years ago Hi @geerlingguy ,
I'm trying to make JMB582 / JMB585 SATA Controller Chipset working with my set-up but without success.
I'm using :
Compute module 4, 1Gb RAM with 16Gb eMMC without WIFI.
This plugged on the "Waveshare Raspberry Pi Compute Module 4 IO Board"
I installed latest raspiOS version and upgraded it.
LSPCI give me this answer :
and this in verbose mode :
dmesg :
Did I missed something ? do you have any idea how I could this this issue?
Thanks for help :)
solstyce
commented
3 years ago Resolved by manually updating Kernel version : https://www.raspberrypi.org/documentation/computers/linux_kernel.html
Result :
grasmax
commented
1 year ago Have successfullly testet this configuration (used as pv controller and local storage):
geerlingguy
commented
1 year ago Testing this on the Raspberry Pi 5:
pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001And more info:
sudo lspci -vvvv
0000:01:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller (prog-if 01 [AHCI 1.0])
Subsystem: JMicron Technology Corp. JMB58x AHCI SATA controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 169
Region 5: Memory at 1b00010000 (32-bit, non-prefetchable) [size=8K]
Expansion ROM at 1b00000000 [virtual] [disabled] [size=64K]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Address: 000000ffffffe000 Data: 0008
Masking: 000000fe Pending: 00000000
Capabilities: [c0] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP+ Rollover- Timeout+ AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [160 v1] Power Budgeting <?>
Capabilities: [1b8 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: LaneErr at lane: 0
Capabilities: [900 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: ahci
Kernel modules: ahciAt PCIe Gen 3.0 speeds, I get a fail on the Pi 5:
[ 8.062038] ata1: found unknown device (class 0)
[ 12.722035] ata1: softreset failed (device not ready)
[ 12.865621] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 12.865634] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 12.865639] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000080/00002000
[ 12.865643] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 18.074038] ata1: found unknown device (class 0)
[ 22.734035] ata1: softreset failed (device not ready)
[ 26.992262] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 26.992271] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 26.992275] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000080/00002000
[ 26.992279] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 28.086038] ata1: found unknown device (class 0)
[ 33.286035] ata1: link is slow to respond, please be patient (ready=0)
[ 35.687176] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 35.687189] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 35.687193] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000080/00002000
[ 35.687198] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 36.098686] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 36.098695] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 36.098699] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000080/00002000
[ 36.098703] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 50.874258] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 50.874268] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[ 50.874272] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00000080/00002000
[ 50.874276] pcieport 0000:00:00.0: [ 7] BadDLLP
[ 57.766035] ata1: softreset failed (device not ready)
[ 57.766042] ata1: limiting SATA link speed to 3.0 Gbps
[ 62.802035] ata1: softreset failed (device not ready)
[ 62.802047] ata1: reset failed, giving up
[ 63.116217] ata2: SATA link down (SStatus 0 SControl 300)And another reboot at PCIe Gen 2.0, all messages:
[ 2.785544] libata version 3.00 loaded.
[ 3.081682] ata1: SATA max UDMA/133 abar m8192@0x1b00010000 port 0x1b00010100 irq 173
[ 3.081691] ata2: SATA max UDMA/133 abar m8192@0x1b00010000 port 0x1b00010180 irq 173
[ 3.081694] ata3: DUMMY
[ 3.081697] ata4: DUMMY
[ 3.081700] ata5: DUMMY
[ 8.442495] ata1: found unknown device (class 0)
[ 13.102490] ata1: softreset failed (device not ready)
[ 18.454494] ata1: found unknown device (class 0)
[ 23.114490] ata1: softreset failed (device not ready)
[ 28.466494] ata1: found unknown device (class 0)
[ 33.666492] ata1: link is slow to respond, please be patient (ready=0)
[ 58.146492] ata1: softreset failed (device not ready)
[ 58.146497] ata1: limiting SATA link speed to 3.0 Gbps
[ 63.182492] ata1: softreset failed (device not ready)
[ 63.182499] ata1: reset failed, giving up
[ 63.496675] ata2: SATA link down (SStatus 0 SControl 300)
PeterSuh-Q3
commented
1 year ago I use Silverstone's SS-ECS07 using JMB585. https://www.silverstonetek.com/en/product/info/expansion-cards/ECS07/
The strange thing is that there is no problem during normal booting. Upon reboot, the following log (dmesg) occurs. It seems almost similar to the log posted above.
The OS is Synology DSM 7 with XPENOLOGY REDPILL.
Will this problem be solved with firmware update? https://www.station-drivers.com/index.php/en/component/remository/func-startdown/5013/lang,en-gb/
[ 2.819555] redpill: module verification failed: signature and/or required key missing - tainting kernel [ 2.846488] PCI host bridge to bus 0001:01 [ 2.851421] pci_bus 0001:01: root bus resource [io 0x0000-0xffff] [ 2.858882] pci_bus 0001:01: root bus resource [mem 0x00000000-0x7fffffffff] [ 2.867376] pci_bus 0001:01: root bus resource [bus 00-ff] [ 2.873980] pci 0001:01:00.0: [1b4b:9215] type 00 class 0x010601 [ 2.881226] pci 0001:01:00.0: Can't map mv9235 registers [ 2.887654] PCI host bridge to bus 0001:02 [ 2.892604] pci_bus 0001:02: root bus resource [io 0x0000-0xffff] [ 2.900063] pci_bus 0001:02: root bus resource [mem 0x00000000-0x7fffffffff] [ 2.908547] pci_bus 0001:02: busn_res: can not insert [bus 02-ff] under domain [bus 00-ff] (conflicts with (null) [bus 01-ff]) [ 2.922288] pci_bus 0001:02: root bus resource [bus 00-ff] [ 2.928892] pci 0001:02:00.0: [8086:1539] type 00 class 0x020000 [ 2.936168] PCI host bridge to bus 0001:03 [ 2.941108] pci_bus 0001:03: root bus resource [io 0x0000-0xffff] [ 2.948566] pci_bus 0001:03: root bus resource [mem 0x00000000-0x7fffffffff] [ 2.957068] pci_bus 0001:03: busn_res: can not insert [bus 03-ff] under domain [bus 00-ff] (conflicts with (null) [bus 01-ff]) [ 2.970798] pci_bus 0001:03: root bus resource [bus 00-ff] [ 2.977414] pci 0001:03:00.0: [8086:1539] type 00 class 0x020000 [ 2.984693] PCI host bridge to bus 0001:00 [ 2.989638] pci_bus 0001:00: root bus resource [io 0x0000-0xffff] [ 2.997106] pci_bus 0001:00: root bus resource [mem 0x00000000-0x7fffffffff] [ 3.005609] pci_bus 0001:00: busn_res: can not insert [bus 00-ff] under domain [bus 00-ff] (conflicts with (null) [bus 01-ff]) [ 3.019349] pci_bus 0001:00: root bus resource [bus 00-ff] ... [ 3.498046] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: id=00e8 [ 3.498079] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=00e8(Receiver ID) [ 3.498080] pcieport 0000:00:1d.0: device [8086:a29a] error status/mask=00000001/00002000 [ 3.498081] pcieport 0000:00:1d.0: [ 0] Receiver Error
I'm currently testing an IOCrest JMB582 storage card.
Testing ongoing. I'm getting a kernel panic during boot whenever the card is plugged in. I've tried increasing the BAR space, and compiling the kernel with AHCI SATA modules.
Kernel panic video:
https://user-images.githubusercontent.com/1296162/106399917-58420080-63d0-11eb-936f-a33bb9d0c72b.mov