search

cisco-system-traffic-generator / trex-core

trex-core site
https://trex-tgn.cisco.com/
Other
1.32k stars 465 forks source link

ERROR in DPDK map - Could not find requested interface #919

Open nexusvista opened 2 years ago

nexusvista commented 2 years ago

Issue and Steps to Reproduce

Hi, I have encountered an issue with initial setup of TRex latest (v2.99).

I have created a loopback connection between eno2 and eno3 with both linkup. Interface eno2 and eno3 were chosen for testing interfaces.

$ sudo ./dpdk_setup_ports.py -t
+----+------+---------+-------------------+-----------------------------------------+---------+----------+----------+
| ID | NUMA |   PCI   |        MAC        |                  Name                   | Driver  | Linux IF |  Active  |
+====+======+=========+===================+=========================================+=========+==========+==========+
| 0  | 0    | 02:00.0 | f4:03:43:3d:75:18 | NetXtreme BCM5719 Gigabit Ethernet PCIe | tg3     | eno1     | *Active* |
+----+------+---------+-------------------+-----------------------------------------+---------+----------+----------+
| 1  | 0    | 02:00.1 |                   | NetXtreme BCM5719 Gigabit Ethernet PCIe | igb_uio |          |          |
+----+------+---------+-------------------+-----------------------------------------+---------+----------+----------+
| 2  | 0    | 02:00.2 |                   | NetXtreme BCM5719 Gigabit Ethernet PCIe | igb_uio |          |          |
+----+------+---------+-------------------+-----------------------------------------+---------+----------+----------+
| 3  | 0    | 02:00.3 | f4:03:43:3d:75:1b | NetXtreme BCM5719 Gigabit Ethernet PCIe | tg3     | eno4     |          |
+----+------+---------+-------------------+-----------------------------------------+---------+----------+----------+

Here is the cfg file with minimal setup.

$ cat /etc/trex_cfg.yaml
- port_limit      : 2
  version         : 2
#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options
  interfaces    : ["02:00.1","02:00.2"] 
  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.
          - ip         : 1.1.1.1
            default_gw : 2.2.2.2
          - ip         : 2.2.2.2
            default_gw : 1.1.1.1

When I tried to run a simple test, the application existed with the error below

$ sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 10
The ports are bound/configured.
Starting  TRex v2.99 please wait  ... 
ERROR in DPDK map 
Could not find requested interface 02:00.1

It seemed that the interfaces are bound to DPDK drivers.

$ sudo ./dpdk_setup_ports.py -s   

Network devices using DPDK-compatible driver
============================================
0000:02:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' drv=igb_uio unused=tg3,vfio-pci,uio_pci_generic
0000:02:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' drv=igb_uio unused=tg3,vfio-pci,uio_pci_generic

Network devices using kernel driver
===================================
0000:02:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:02:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

But now the problem is I can not see the eno2 and eno3 from ifconfig, nor from TRex.

$ sudo ./t-rex-64 -i
Starting Scapy server..... Scapy server is started
The ports are bound/configured.
Starting  TRex v2.99 please wait  ... 
ERROR in DPDK map 
Could not find requested interface 02:00.1
Killing Scapy server... Scapy server is killed

When I tried to return the two interfaces back to Linux kernel, it did not show any interfaces.

$ sudo ./dpdk_setup_ports.py -l
Could not determine interfaces information. Try to run manually: sudo ./t-rex-64 --dump-interfaces
$ sudo ./t-rex-64 --dump-interfaces

Starting  TRex v2.99 please wait  ... 
Showing interfaces info.
(none output here)

Versions

OS DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

$ uname -r 4.15.0-122-generic // could this be a problem?

$ ldd --version ldd (Ubuntu GLIBC 2.27-3ubuntu1.5) 2.27

TRex version: v2.99

hhaim commented 2 years ago

@nexusvista netXstream BMC is not part of our regression setups as a result the support for this NIC could be broken, You could try older versions or try to fix this issue

jgato commented 2 years ago

I am having similar issues, how do I know if my cards are supported? (Ethernet Controller E810-C for SFP)

pkramer509 commented 1 year ago

I have the same issue with the BCM5720, both on Centos 7.6 and Ubuntu 22.04 LTS.

I went ahead and tested with v3.00, v2.99, v2.90 and v2.50. All have the same issue.

dylan2intel commented 1 year ago

The same issue reproduced on Ethernet Controller E810-C for QSFP, all failed from v2.97 to v3.00

root@benchmark-n28hx:/opt/trex-v3.00# $TREX_HOME/t-rex-64 --prom -c 15 -i
Starting Scapy server.... Scapy server is started
Trying to bind to vfio-pci ...
/usr/local/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:99:00.0 0000:99:00.1
The ports are bound/configured.
Starting  TRex v3.00 please wait  ...
EAL: Requested device 0000:99:00.0 cannot be used
EAL: Requested device 0000:99:00.1 cannot be used
EAL: Bus (pci) probe failed.
ERROR in DPDK map
Could not find requested interface 0000:99:00.0
Killing Scapy server... Scapy server is killed
root@benchmark-n28hx:/opt/trex-v3.00# /usr/local/bin/python3 dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================
0000:98:00.0 'Ethernet Controller E810-C for QSFP' drv=vfio-pci unused=ice
0000:99:00.0 'Ethernet Controller E810-C for QSFP' drv=vfio-pci unused=ice
0000:99:00.1 'Ethernet Controller E810-C for QSFP' drv=vfio-pci unused=ice

Network devices using kernel driver
===================================
0000:16:00.0 'Ethernet Controller X710 for 10GbE SFP+' if= drv=i40e unused=vfio-pci
0000:16:00.1 'Ethernet Controller X710 for 10GbE SFP+' if= drv=i40e unused=vfio-pci
0000:98:00.1 'Ethernet Controller E810-C for QSFP' if= drv=ice unused=vfio-pci

Other network devices
=====================
<none>
wittling commented 1 year ago

I had this problem, when I moved my Trex VMs over to a new VMWare environment (cluster), and put SR-IOV adaptors on the VMs. When I booted the VMs up, the igb_uio.ko drivers were not even loaded. I did a manual modprobe on uio, and then used dpdk_bind --bind=igb_uio 0000:03:00.0 (same with another PCI address). They bound just find, so that when I ran dpdk_status they both showed up using igb_uio drivers. But when I tried to start Trex I got the same exact error: Starting TRex v2.99 please wait ... ERROR in DPDK map Could not find requested interface 02:00.1

I am now putting VMXNET3 nics on the VM instead of SR-IOV nics. Hopefully mac addresses will stay the same. Then I can retry this. I suspect the issue is that you cannot run DPDK on SR-IOV (passthrough) adaptors.

nexusvista commented 1 year ago

Dear all, may I ask if anyone has any success in using SolarFlare XtremeScale X2542+ QSFP28 PCIe Dual Port (https://www.xilinx.com/content/dam/xilinx/publications/solarflare/quick-start-guides/SF-121025-CD-4_X2542_online_Quick_Start_Guide.pdf) for TRex please?

yogisha52 commented 1 year ago

Hello everyone, Could someone tell me if they have managed to get a Netronome smartNIC to work?

acolyte-py commented 1 year ago

Dear all, I have a similar problem, you can see from the "status" that the port is there. But when trying to run a script (if necessary I can give an example) shows that the port can not be used and eventually it says no port at all. Who finally solved this issue, or as there is no such solution - because the issue is still open?

traceback:

[root@localhost traffic_generator]# ./trex.sh --safe-mode-support=1
The ports are bound/configured.
Starting  TRex v3.02 please wait  ... 
ice_load_pkg(): failed to search file path

ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode
EAL: Requested device 0000:01:00.0 cannot be used
ice_load_pkg(): failed to search file path

ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode
EAL: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
ERROR in DPDK map 
Could not find requested interface 0000:01:00.0

log-status:

[admin@localhost ~]$ dpdk-devbind.py --status-dev=net

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=uio_pci_generic unused=ice
0000:01:00.1 'Ethernet Controller E810-C for QSFP 1592' drv=uio_pci_generic unused=ice

Network devices using kernel driver
===================================
0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb unused=uio_pci_generic *Active*
0000:07:00.0 'I210 Gigabit Network Connection 1533' if=eno2 drv=igb unused=uio_pci_generic

p.s. 1 more log:

[root@localhost traffic_generator]# dpdk-testpmd 
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 1048576 kB hugepages reported
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.0 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (double VLAN mode)
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:01:00.1 (socket 0)
ice_load_pkg_type(): Active package is: 1.3.30.0, ICE OS Default Package (double VLAN mode)
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: create a new mbuf pool <mb_pool_0>: n=267456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
ice_set_rx_function(): Using AVX2 Vector Rx (port 0).
Port 0: 6C:B3:11:21:AD:66
Configuring Port 1 (socket 0)
ice_set_rx_function(): Using AVX2 Vector Rx (port 1).
Port 1: 6C:B3:11:21:AD:67
Checking link statuses...
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
Press enter to exit

Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye..
acolyte-py commented 1 year ago

Dear all, I would like to share with you the news how exactly I managed to fix this error and run trex in full. To begin with, I came across this material dosc in it I made my conclusions and next steps: 1) Downloaded a new version of ice.pkg link pkg 2) Deleted the old one and made a symbolic link to the new ice.pkg 3) Added this package to /lib/firmware/updates/intel/ice/ddp/ice.pkg.

2 step list commands (root):

cd /lib/firmware/intel/ice/ddp
rm ice.pkg.xz
ln -s ice_comms-1.3.45.0.pkg ice.pkg.xz

Started trex again and there was no more error

OS settings: uname -r: 6.0.7-301.fc37.x86_64

cat /etc/os-release | grep VERSION: PRETTY_NAME="Fedora Linux 37 (Server Edition)" VERSION="37 (Server Edition)" VERSION_ID=37 VERSION_CODENAME="" REDHAT_BUGZILLA_PRODUCT_VERSION=37 REDHAT_SUPPORT_PRODUCT_VERSION=37

ldd version: ldd (GNU libc) 2.36

TRex version: v3.02

Decision Date: 18.10.2023

dosarudaniel commented 10 months ago

Hi all, is there any MANA NIC support for trex v3.04 ?

Because I am working on a MANA NIC setup and even though I could see the interfaces here:

/root/v3.04$ sudo ./dpdk_setup_ports.py -t
+----+------+--------------+-------------------+-------------+--------+------------+--------+
| ID | NUMA |     PCI      |        MAC        |    Name     | Driver |  Linux IF  | Active |
+====+======+==============+===================+=============+========+============+========+
| 0  | 0    | 6abd:00:02.0 | 00:15:5d:06:23:04 | Device 00ba | mana   | enP27325s2 |        |
+----+------+--------------+-------------------+-------------+--------+------------+--------+
| 1  | 0    | 95ee:00:02.0 | 00:15:5d:06:23:0b | Device 00ba | mana   | enP38382s3 |        |
+----+------+--------------+-------------------+-------------+--------+------------+--------+
/root/v3.04$ cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['6abd:00:02.0', '95ee:00:02.0']
  port_info:
      - ip: 1.1.1.1
        default_gw: 2.2.2.2
      - ip: 2.2.2.2
        default_gw: 1.1.1.1

  platform:
      master_thread_id: 0
      latency_thread_id: 15
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14]

I could not start the trex server:

/root/v3.04$ sudo ./_t-rex-64-debug -i
Starting  TRex v3.04 please wait  ...
ERROR in DPDK map
Could not find requested interface 6abd:00:02.0