search

NVIDIA / l2fwd-nv

l2fwd-nv provides an example of how to leverage your DPDK network application with the NVIDIA GPUDirect RDMA techonology.
Apache License 2.0
65 stars 20 forks source link

The performance is not reach to 100Gbps #6

Closed muthuramanecs03g closed 2 years ago

muthuramanecs03g commented 2 years ago

I am not able to reach to 100Gbps and only get 55Gbps.

The Linux dmesg log reports the maximum as 63Gb/s only like below

[10380.315885] mlx5_core 0000:b3:00.1: 63.008 Gb/s available PCIe bandwidth, limited by 8 GT/s x8 link at 0000:b2:00.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)

Could you please give some comments on my configuration? and direct me on how to improve the performance?

System Information:

Host: 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux CPU: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz Mellanox PN: MCX623106AN-CDAT Mellanox NIC FW: 22.32.2004 DPDK: v21.11-rc4 OFED: MLNX_OFED_LINUX-5.5-1.0.3.2:

DPDK Testmd:

sudo ./dpdk-testpmd -l 2-10 --main-lcore=2 -a b3:00.0 -- --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024 --rxd=1024 --mbcache=512 --rxq=4 --txq=4 --forward-mode=txonly -i --nb-cores=8 --txonly-multi-flow EAL: Detected CPU lcores: 24 EAL: Detected NUMA nodes: 1 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:b3:00.1 (socket 0) TELEMETRY: No legacy callbacks, legacy socket not created Set txonly packet forwarding mode Interactive-mode selected testpmd: create a new mbuf pool : n=278528, size=2048, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0) Port 0: 0C:42:A1:B1:C1:D1 Checking link statuses... Done

testpmd> set txpkts 1024 testpmd> start txonly packet forwarding - ports=1 - cores=4 - streams=4 - NUMA support enabled, MP allocation mode: native Logical Core 3 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 Logical Core 4 (socket 0) forwards packets on 1 streams: RX P=0/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00 Logical Core 5 (socket 0) forwards packets on 1 streams: RX P=0/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00 Logical Core 6 (socket 0) forwards packets on 1 streams: RX P=0/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00

txonly packet forwarding packets/burst=64 packet len=1024 - nb packet segments=1 nb forwarding cores=8 - nb forwarding ports=1 port 0: RX queue number: 4 Tx queue number: 4 Rx offloads=0x0 Tx offloads=0x10000 RX queue: 0 RX desc=1024 - RX free threshold=64 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=1024 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x10000 - TX RS bit threshold=0 testpmd> stop Telling cores to stop... Waiting for lcores to finish...

------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0 ------- RX-packets: 0 TX-packets: 239418112 TX-dropped: 5639512704

------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 0/Queue= 1 ------- RX-packets: 0 TX-packets: 239418240 TX-dropped: 5641563328

------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 0/Queue= 2 ------- RX-packets: 0 TX-packets: 239418176 TX-dropped: 5642059072

------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 0/Queue= 3 ------- RX-packets: 0 TX-packets: 239418112 TX-dropped: 5629728832

---------------------- Forward statistics for port 0 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 957672640 TX-dropped: 22552863936 TX-total: 23510536576

+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 957672640 TX-dropped: 22552863936 TX-total: 23510536576 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Mellanox perf log:

  tx_vport_unicast_packets: 6,695,119
    tx_vport_unicast_bytes: 6,855,801,856 Bps    = 54,846.41 Mbps      
            tx_packets_phy: 6,695,110
              tx_bytes_phy: 6,882,575,972 Bps    = 55,060.60 Mbps      
            tx_prio0_bytes: 6,882,515,512 Bps    = 55,060.12 Mbps      
          tx_prio0_packets: 6,695,053
                      UP 0: 55,060.12            Mbps = 100.00%
                      UP 0: 6,695,053            Tran/sec = 100.00%

The lspci log:

b3:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] Subsystem: Mellanox Technologies MT28841 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 66 NUMA node: 0 Region 0: Memory at e4000000 (64-bit, prefetchable) [size=32M] Expansion ROM at e1000000 [disabled] [size=1M] Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [48] Vital Product Data Product Name: ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe 4.0 x16, No Crypto
Read-only fields: [PN] Part number: MCX623106AN-CDAT [EC] Engineering changes: A6 [V2] Vendor specific: MCX623106AN-CDAT [SN] Serial number: MT2006X19556 [V3] Vendor specific: de46691c904cea1180000c42a11dd12a [VA] Vendor specific: MLX:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0:MODL=CX623106A [V0] Vendor specific: PCIeGen4 x16 [RV] Reserved: checksum good, 1 byte(s) reserved End Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=0 offset=00003000 Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 04, GenCap+ CGenEn+ ChkCap+ ChkEn+ Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 VF offset: 2, stride: 1, Device ID: 101e Supported Page Size: 000007ff, System Page Size: 00000001 Region 0: Memory at 00000000e6800000 (64-bit, prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [1c0 v1] #19 Capabilities: [230 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [320 v1] #27 Capabilities: [370 v1] #26 Capabilities: [420 v1] #25 Kernel driver in use: mlx5_core Kernel modules: mlx5_core

Log ibstat:

CA 'mlx5_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.32.2004 Hardware version: 0 Node GUID: 0x0c42a103001dd12a System image GUID: 0x0c42a103001dd12a Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x0e42a1fffe1dd12a Link layer: Ethernet CA 'mlx5_1' CA type: MT4125 Number of ports: 1 Firmware version: 22.32.2004 Hardware version: 0 Node GUID: 0x0c42a103001dd12b System image GUID: 0x0c42a103001dd12a Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x0e42a1fffe1dd12b Link layer: Ethernet

muthuramanecs03g commented 2 years ago

Issue not seen in PCIe Gen 3.0 x16, and closing this issue