Dmesg Reads that PCIe Bandwidth is limited to 2.5 GT/s PCIe x4

[wizardNoel]

Hi,

In the dmesg log below it looks like my SSD is limited to 2.5GT/s. Is this expected?

My setup is the discontinued FPGA driver card(Non Gen4) with Ultazed carrier card. To build the project I used Petalinux 2022.1 and in efforts to save disk space for Vivado I did a hack where I used Vivado 2021.2 on the 2020.2 release tag.

`.983331] xilinx-xdma-pcie 400000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@a0000000 ranges:

.990805] xilinx-xdma-pcie 400000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@a0000000, using [bus 00-ff]

.001711] xilinx-xdma-pcie 400000000.axi-pcie: MEM 0x00a0000000..0x00afffffff -> 0x00a0000000

.010823] xilinx-xdma-pcie 400000000.axi-pcie: Using MSI Decode mode

.017332] xilinx-xdma-pcie 400000000.axi-pcie: PCIe Link is UP

.023341] xilinx-xdma-pcie 400000000.axi-pcie: PCI host bridge to bus 0000:00

.030505] pci_bus 0000:00: root bus resource [bus 00-ff]

.035952] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xafffffff]

.042811] pci 0000:00:00.0: [10ee:9134] type 01 class 0x060400

.050230] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring

.056814] pci 0000:01:00.0: [1987:5012] type 00 class 0x010802

.062712] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]

.069639] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0

.085681] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

.090894] pci 0000:00:00.0: BAR 8: assigned [mem 0xa0000000-0xa00fffff]

.097630] pci 0000:01:00.0: BAR 0: assigned [mem 0xa0000000-0xa0003fff 64bit]

.104908] pci 0000:00:00.0: PCI bridge to [bus 01]

.109825] pci 0000:00:00.0: bridge window [mem 0xa0000000-0xa00fffff]

.116643] xilinx-xdma-pcie 500000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@b0000000 ranges:

.125396] xilinx-xdma-pcie 500000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@b0000000, using [bus 00-ff]

.136301] xilinx-xdma-pcie 500000000.axi-pcie: MEM 0x00b0000000..0x00bfffffff -> 0x00b0000000

.145417] xilinx-xdma-pcie 500000000.axi-pcie: Using MSI Decode mode

.151912] xilinx-xdma-pcie 500000000.axi-pcie: PCIe Link is DOWN

.158100] xilinx-xdma-pcie 500000000.axi-pcie: PCI host bridge to bus 0001:00

.165272] pci_bus 0001:00: root bus resource [bus 00-ff]

.170720] pci_bus 0001:00: root bus resource [mem 0xb0000000-0xbfffffff]

.177574] pci 0001:00:00.0: [10ee:9134] type 01 class 0x060400

.184933] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring

.192803] pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01

.198060] pci 0001:00:00.0: PCI bridge to [bus 01]
`

[fpgadeveloper]

What SSD are you using? It should at least support PCIe Gen3 to achieve 8GT/s. The UltraZed-EV can support Gen3 as long as the SSD can support it.

[wizardNoel]

What SSD are you using? It should at least support PCIe Gen3 to achieve 8GT/s. The UltraZed-EV can support Gen3 as long as the SSD can support it.

Hi Jeff,

Im using this one have a look. https://sabrent.com/collections/internal-memory/products/sb-rktq-500

Its indeed Gen3.

[fpgadeveloper]

Looks like it should work. Do you have any other Gen3 SSDs that you could test? It's possible that this SSD requires some adjustments to the GT settings in the design. It might be having trouble during the link negotiation process and stepping down to Gen1.

[wizardNoel]

Hi Jeff there are also some host bridge errors

`.983331] xilinx-xdma-pcie 400000000.axi-pcie: host bridge /amba_pl@0/axi-pcie@a0000000 ranges:

.990805] xilinx-xdma-pcie 400000000.axi-pcie: No bus range found for /amba_pl@0/axi-pcie@a0000000, using [bus 00-ff]

.001711] xilinx-xdma-pcie 400000000.axi-pcie: MEM 0x00a0000000..0x00afffffff -> 0x00a0000000

.010823] xilinx-xdma-pcie 400000000.axi-pcie: Using MSI Decode mode

.017332] xilinx-xdma-pcie 400000000.axi-pcie: PCIe Link is UP

.023341] xilinx-xdma-pcie 400000000.axi-pcie: PCI host bridge to bus 0000:00

.030505] pci_bus 0000:00: root bus resource [bus 00-ff]

.035952] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xafffffff]

.042811] pci 0000:00:00.0: [10ee:9134] type 01 class 0x060400

.050230] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring`

[fpgadeveloper]

That's expected, you can ignore those messages.

[wizardNoel]

Master branch is backwards compatible for discontinued FPGA drive correct?

[fpgadeveloper]

No. If you are using discontinued FPGA Drive (PN: OP044) then you need to go back to an earlier release. Same for PN: OP047.

[wizardNoel]

That's pre Ultra Zed era?

[fpgadeveloper]

Which one are you using? OP044 or OP047 (single SSD)?

[wizardNoel]

OP047 REV D, dual SSD.

Tag 2020.2 has UZEV but mentions GEN 4 fpga drive FMC.

Tag 2019 doesn't have UZEV but doesn't mention GEN 4 fpga drive FMC

Can I get by with tag v2020.2 ?

[fpgadeveloper]

That one will work fine with the latest release. The ones that don't work with the later releases are OP044 and the version of OP047 with only one SSD slot.

[wizardNoel]

Is the downgraded speed normal behavior? sudo lspci -vv -s 0000:01:00.0 LnkSta: Speed 2.5GT/s (downgraded), Width x4 (ok)

What is the expected dmesg output for Gen3 SSD?

Do you observe the same? Maybe I need some Kernel Performance configs enabled?

[fpgadeveloper]

As I mentioned earlier, the GT settings in the design are not necessarily optimal for all SSDs. It is normal behavior for some SSDs to downgrade the link speed if they fail the negotiation process at higher speeds.

Using 2x Sabrent 1TB Rocket NVMe PCIe M.2 2230 SSDs (PCIe Gen4 capable), FPGA Drive FMC Gen4 (OP063) and UltraZed-EV Carrier, these are the messages that I get:

[    3.067686] pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)

0000:01:00.0 Non-Volatile memory controller: Phison Electronics Corporation Device 5021 (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Phison Electronics Corporation Device 5021
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 71
        Region 0: Memory at a0000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (downgraded), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

If you do not have any other SSDs to try, then I suggest you try changing the GT settings:

CONFIG.ins_loss_profile {Chip-to-Chip}

https://github.com/fpgadeveloper/fpga-drive-aximm-pcie/blob/master/Vivado/src/bd/bd_zynqmp.tcl

Or simply open the block diagram in Vivado and double click on the XDMA IP. Then you will find the GT settings in one of the tabs. You'll have to play around with different values until you find what works for your SSD.

This is no longer related to the design, please don't write any further messages on this issue. For further support, contact me by using the contact form.