search

pktgen / Pktgen-DPDK

DPDK based packet generator
Other
389 stars 119 forks source link

Did not find any ports to use #245

Closed Aniurm closed 7 months ago

Aniurm commented 7 months ago

What's the problem

I create a VM and attach a VF(ConnectX-5 SRIOV) to it using virt-manager (Reference)

VF should be configured successfully, as I can ping other IPs with it.

I compiled DPDK successfully following this Guide

~/dpdk-23.11$ cat VERSION
23.11.0

Then I compiled DPDK-Pktgen following this Guide

I tried to run pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fbd3eb7d062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fbd3eb552a0]
2: ./pktgen (55a87c076000+0x36175) [55a87c0ac175]
3: ./pktgen (55a87c076000+0xb587) [55a87c081587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fbd3e776083]
5: ./pktgen (55a87c076000+0xbb7e) [55a87c081b7e]
Aborted

Info of my system

$ ./dpdk-23.11/usertools/dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci *Active*
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' if=enp7s0np0 drv=mlx5_core unused=vfio-pci

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

Misc (rawdev) devices using kernel driver
=========================================
0000:03:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci
0000:04:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci

No 'Regex' devices detected
===========================

No 'ML' devices detected
========================
$ grep -i huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1023
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB
$ uname -r
5.4.0-174-generic
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

What I tried

Change driver

Reference: This Github Issue

$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device
Error: unbind failed for 0000:07:00.0 - Cannot open /sys/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'
$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 0000:07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device

Error: No such device - I don't know why.

Add parameter to pktgen

Reference: Stack Overflow

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -d librte_net_mlx5.so -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:a2d3) device: 0000:07:00.0 (socket -1)

====== Pktgen got a Segment Fault

Obtained 12 stack frames.
./pktgen(+0x29535) [0x558259321535]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9d49dc1420]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1ca8c) [0x7f9d494dea8c]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1e2e3) [0x7f9d494e02e3]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x15e0b0) [0x7f9d496200b0]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.24(+0xa7cc) [0x7f9d4949b7cc]
/usr/local/lib/x86_64-linux-gnu/librte_bus_pci.so.24(+0x4500) [0x7f9d49b7f500]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(rte_bus_probe+0x33) [0x7f9d49fc5313]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(+0x15654) [0x7f9d49fc1654]
./pktgen(+0xaf38) [0x558259302f38]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f9d49bdd083]
./pktgen(+0xbb7e) [0x558259303b7e]

Got a Segment Fault

Set CONFIG_RTE_LIBRTE_MLX5_PMD=y

Reference: Nvidia Forum

~/dpdk-23.11$ cat config/common_base
CONFIG_RTE_LIBRTE_MLX5_PMD=y

I added config/common_base in DPDK folder and compiled DPDK.

I don't know whether this work or not.

Reinstall MLNX_OFED driver

I download MLNX_OFED_LINUX-24.01-0.3.3.1-ubuntu20.04-x86_64 from Nvidia website and install:

sudo ./mlnxofedinstall --upstream-libs --dpdk --force

But it didn't solve my problem.

Could anyone help me resolve this bug that has been troubling me for a long time?
KeithWiles commented 7 months ago

It looks like dpdk-devbind.py is not working.

Make sure you have GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt" add to your /etc/default/grub file, then you must reboot. Here is the changes I made.

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
#GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt"

I commented out the first GRUB_CMDLINE_LINUX and created the second one. Please verify this is correct for your OS and distro.

Aniurm commented 7 months ago

Thank you for your help!

I follow your instructions:

Unfortunately it didn't let dpdk-devbind.py work, then I refer to Stack Overflow

$ modprobe vfio enable_unsafe_noiommu_mode=1
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

After this, I can use dpdk-devbind.py to change the driver of my VF to vfio-pci.

Network devices using DPDK-compatible driver 
============================================
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' drv=vfio-pci unused=mlx5_core

But I still cannot use Pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fdbebd60062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fdbebd382a0]
2: ./pktgen (56525adea000+0x36175) [56525ae20175]
3: ./pktgen (56525adea000+0xb587) [56525adf5587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fdbeb959083]
5: ./pktgen (56525adea000+0xbb7e) [56525adf5b7e]
Aborted

I think I have enough memory for Pktgen?

$ grep -i huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1023
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB
KeithWiles commented 7 months ago

Sorry, not going to be much help as I do not run Pktgen inside a VM, but I run it on bare metal linux based machines.

I run with 16K 2M huge pages, but I believe 1024 2M hugepages should work, besides you are still getting the error no port found. This means DPDK is not finding or detecting this NIC, could be a DPDK driver issues of something.

If you can try a different NIC or verify the version of DPDK you are using supports that NIC.

This problem is more of a DPDK/NIC problem then a Pktgen problem IMO. You can also try the different DPDK examples or testpmd to verify these work and if so then we can see if we can get Pktgen to work. The problem will be is I do not have this NIC and will not be able to debug the problem :-(

Aniurm commented 7 months ago

Thank you for your help and patience, I really appreciate it. I'll try to solve this, once I make it work I'll update in this issue so I can help others

Aniurm commented 7 months ago

testpmd can detect VF and show its info:

# dpdk-testpmd -l 8-15 -n 4 -a 07:00.0 -- --rxq=2 --txq=2 -i
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:a2d3) device: 0000:07:00.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.
testpmd: Flow tunnel offload support might be limited or unavailable on port 0
testpmd: create a new mbuf pool <mb_pool_0>: n=203456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
Port 0: B2:13:6C:40:86:92
Checking link statuses...
Done
testpmd>
testpmd> show port info 0

********************* Infos for port 0  *********************
MAC address: B2:13:6C:40:86:92
Device name: 07:00.0
Driver name: mlx5_pci
Firmware-version: 18.24.0302
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 100 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 2
Supported RSS offload flow types:
  ipv4  ipv4-frag  ipv4-tcp  ipv4-udp  ipv4-other  ipv6
  ipv6-frag  ipv6-tcp  ipv6-udp  ipv6-other  ipv6-ex
  ipv6-tcp-ex  ipv6-udp-ex  esp  l4-dst-only  l4-src-only
  l3-dst-only  l3-src-only
Minimum size of RX buffer: 32
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 65280
Current number of RX queues: 2
Max possible RX queues: 1024
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 2
Max possible TX queues: 1024
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x10( FLOW_SHARED_OBJECT_KEEP )
Switch name: 07:00.0
Switch domain Id: 0
Switch Port Id: 65535
Device error handling mode: none
Device private info:
  none
Aniurm commented 7 months ago

Finally I solved this. I think this problem is related to shared library about MLX5 PMD or the version of DPDK. What I have done to fix this:

Now I can run Pktgen with option -d librte_net_mlx5.so

I think in the first time I compiled DPDK, I didn't install MLNX_OFED_LINUX-5 at that time, so DPDK was compiled without libraries about MLX5.

KeithWiles commented 7 months ago

Thanks, I was thinking along the same lines meaning that DPDK is not detecting the NIC. Pktgen asks DPDK the number of ports available (via rte_eth_dev_count_avail() routine) if it states zero then Pktgen can't do much. I was thinking DPDK was not detecting the NIC and was not able to find a driver.

Glad you figured it out.