search

pktgen / Pktgen-DPDK

DPDK based packet generator
Other
384 stars 119 forks source link

ICMP with not response found #252

Open lucascgo opened 5 months ago

lucascgo commented 5 months ago

I have a setup with two VFs binded to DPDK. I want to do a simple ping test using one of this VFs (port 0), inside my network, like this:

+-------------+        +-------------+      +-------------+      
| DPDK Port 0 |--------|   Gateway   |------|    DUT      |      
| 10.60.1.45  |        |  10.60.1.1  |      | 10.60.1.151 |      
+-------------+        +-------------+      +-------------+

Below the pktgen startup: 

$ sudo pktgen -l 0-1 -n 2 -- -P -T -N -m "1.[0-1]"                                                                                                    

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.                                                                                                                  
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<                                                                                                                         

EAL: Detected CPU lcores: 16                                                                                                                                                           
EAL: Detected NUMA nodes: 1                                                                                                                                                            
EAL: Detected shared linkage of DPDK                                                                                                                                                   
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket                                                                                                                                  
EAL: Selected IOVA mode 'VA'                                                                                                                                                           
EAL: VFIO support initialized                                                                                                                                                          
EAL: Using IOMMU type 1 (Type 1)                                                                                                                                                       
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:02.0 (socket 0)                                                                                                            
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:03.0 (socket 0)                                                                                                            
TELEMETRY: No legacy callbacks, legacy socket not created                                                                                                                              
Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio                                                                                                                                    
    0: net_bnxt        0      0   14e4:16dc/0000:65:02.0                                                                                                                               
    1: net_bnxt        0      0   14e4:16dc/0000:65:03.0                                                                                                                               

Initialize Port 0 -- RxQ 1, TxQ 1                                                                                                                                                      
   Src MAC 62:52:b3:8c:eb:0f <Promiscuous is Enabled>                                                                                                                                  
Initialize Port 1 -- RxQ 1, TxQ 1                                                                                                                                                      
   Src MAC 8a:82:0e:c2:bf:9e <Promiscuous is Enabled>

Below the pktgen running: 

| Ports 0-1 of 2   <Main Page>  Copyright(c) <2010-2023>, Intel Corporation
  Port:Flags        : 0:PE-I---      Single 1:P------      Single
Link State          :         <UP-10000-FD>         <UP-10000-FD>     ---Total Rate---
Pkts/s Rx           :                     1                     0                    1
       Tx           :                     1                     0                    1
MBits/s Rx/Tx       :                   0/0                   0/0                  0/0
Pkts/s Rx Max       :                     4                     4                    5
       Tx Max       :                     4                     0                    4
Broadcast           :                     0                     0
Multicast           :                  3648                 14144
Sizes 64            :                  3520                 19008
      65-127        :                  9472                  1408
      128-255       :                     0                     0
      256-511       :                     0                   704
      512-1023      :                     0                     0
      1024-1518     :                     0                     0
Runts/Jumbos        :                 640/0                   0/0
ARP/ICMP Pkts       :             4032/8704               13440/0
Errors Rx/Tx        :                   0/0                   0/0
Total Rx Pkts       :                   212                   330
      Tx Pkts       :                   144                     0
      Rx/Tx MBs     :                   0/0                   0/0
TCP Flags           :                .A....                .A....
TCP Seq/Ack         :           74616/74640           74616/74640
Pattern Type        :               abcd...               abcd...
Tx Count/% Rate     :         Forever /100%         Forever /100%
Pkt Size/Rx:Tx Burst:           64 / 64: 64           64 / 64: 64
TTL/Port Src/Dest   :        64/ 1234/ 5678        64/ 1234/ 5678
Pkt Type:VLAN ID    :       IPv4 / TCP:0001       IPv4 / TCP:0001
802.1p CoS/DSCP/IPP :             0/  0/  0             0/  0/  0
VxLAN Flg/Grp/vid   :      0000/    0/    0      0000/    0/    0
IP  Destination     :           10.60.1.151           192.168.0.1
    Source          :         10.60.1.45/24        192.168.1.1/24
MAC Destination     :     52:54:00:80:1f:bd     62:52:b3:8c:eb:0f
    Source          :     62:52:b3:8c:eb:0f     8a:82:0e:c2:bf:9e
NUMA/Vend:ID/PCI    :0/14e4:16dc/0000:65:020/14e4:16dc/0000:65:03.0
-- Pktgen 24.03.1 (DPDK 24.07.0-rc0)  Powered by DPDK  (pid:404102) -----------

Pktgen:/> set 0 src ip 10.60.1.45/24
Pktgen:/> set 0 src mac 62:52:b3:8c:eb:0f
Pktgen:/> set 0 dst ip 10.60.1.151
Pktgen:/> set 0 dst mac 52:54:00:80:1f:bd
Pktgen:/> enable 0 icmp
Pktgen:/> enable 0 process
Pktgen:/> start 0 arp request
Pktgen:/> ping4 0
Pktgen:/>

The DUT do not receive the response:

$ ping -I ens10 10.60.1.45
PING 10.60.1.45 (10.60.1.45) from 10.60.1.151 ens10: 56(84) bytes of data.
--- 10.60.1.45 ping statistics ---
1487 packets transmitted, 0 received, 100% packet loss, time 1521653ms

Looking with tcpdump, I received the packets, but they appear to be incomplete:

22:22:29.644258 IP 10.60.1.151 > 10.60.1.45: ICMP echo request, id 265, seq 101, length 64
22:22:29.644547 IP 10.60.1.45 > 10.60.1.151: ICMP echo reply, id 265, seq 101, length 64
22:22:30.668310 IP 10.60.1.151 > 10.60.1.45: ICMP echo request, id 265, seq 102, length 64
22:22:30.668626 IP 10.60.1.45 > 10.60.1.151: ICMP echo reply, id 265, seq 102, length 64
22:22:31.692325 IP 10.60.1.151 > 10.60.1.45: ICMP echo request, id 265, seq 103, length 64
22:22:31.692602 IP 10.60.1.45 > 10.60.1.151: ICMP echo reply, id 265, seq 103, length 64
22:22:32.716315 IP 10.60.1.151 > 10.60.1.45: ICMP echo request, id 265, seq 104, length 64
22:22:32.716840 IP 10.60.1.45 > 10.60.1.151: ICMP echo reply, id 265, seq 104, length 64
22:22:33.740396 IP 10.60.1.151 > 10.60.1.45: ICMP echo request, id 265, seq 105, length 64
22:22:33.740752 IP 10.60.1.45 > 10.60.1.151: ICMP echo reply, id 265, seq 105, length 64

The Wireshark: Screenshot from 2024-04-24 19-27-23

What could it be?

KeithWiles commented 5 months ago

Maybe wrong here, but the Gateway needs two ports and each port needs to be on a different subnet to allow the gateway to route between the two networks, which also means the DPDK and DUT needs to be on different subnets. I also assume the three devices are the only three connected and we do not have a switch connecting all three devices. Maybe the gateway is really a bridge or L2 switch then this will be fine.

10.60.1.45/24 <==> 10.60.1.1/24 GW 10.60.2.1/24 <===> 10.60.2.151/24

It appears the the ping response is working, but I see a lot of Runts on port 0. I see the ICMP counter is 8704 does that counter increase when doing a ping4?

If you stop DPDK rebind the port to the linux kernel and use ping from that machine does ping work?

lucascgo commented 5 months ago

Hi @KeithWiles, thanks for your reply.

I wrote gateway, but is a real switch. The infrastructure is like this:

+-------------+       +-------------+
| DPDK Port 0 |       | DUT Kernel  |
| 10.60.1.45  |       | 10.60.1.151 |
+-------------+       +-------------+
       |                     |       
       |        VLAN         |       
+-----------------------------------+
|               SWITCH              |
|             10.60.1.1             |
+-----------------------------------+

Yes, the ICMP Pkts counter increase with ping4 0 command. And the Runts appear only in the first time when I change from Kernel to DPDK and run the Pktgen.

If I rebind to the linux kernel the machine with DPDK:

$ sudo dpdk-devbind.py -b bnxt_en 0000:65:02.0 0000:65:03.0
$ sudo ip addr add 10.60.1.45/24 dev ens2f0v0

And I do the ping from DUT:

$ ping -I ens10 10.60.1.45
PING 10.60.1.45 (10.60.1.45) from 10.60.1.151 ens10: 56(84) bytes of data.
64 bytes from 10.60.1.45: icmp_seq=1 ttl=64 time=0.309 ms
64 bytes from 10.60.1.45: icmp_seq=2 ttl=64 time=0.261 ms
64 bytes from 10.60.1.45: icmp_seq=3 ttl=64 time=0.385 ms
64 bytes from 10.60.1.45: icmp_seq=4 ttl=64 time=0.399 ms

The funny thing is that it wasn't necessary the command sudo ip link set ens2f0v0 up to ping works. And the interface is DOWN on system:

$ ip -c a | grep ens
3: ens2f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.60.1.44/24 scope global ens2f0np0
4: ens2f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.30.1.44/24 scope global ens2f1np1
365: ens2f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    inet 10.60.1.45/24 scope global ens2f0v0
366: ens2f1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

The interfaces:

┌───────────┬──────────────┬───────────────────────┐
│ ens2f0np0 │ PF1          │ Switch VLAN 10.60.1.x │
├───────────┼──────────────┼───────────────────────┤
│ ens2f1np1 │ PF2          │ Switch VLAN 10.30.1.x │
├───────────┼──────────────┼───────────────────────┤
│ ens2f0v0  │ VF1 from PF1 │ Switch VLAN 10.60.1.x │
├───────────┼──────────────┼───────────────────────┤
│ ens2f1v0  │ VF2 from PF2 │ Switch VLAN 10.30.1.x │
└───────────┴──────────────┴───────────────────────┘
KeithWiles commented 5 months ago

Can you send me the dump.pcap file you are showing above or one like it?

lucascgo commented 5 months ago

Of course, here: dumps.tar.gz

KeithWiles commented 5 months ago

Looks like the checksum is wrong in the reply.

lucascgo commented 5 months ago

And why this wrong checksum? Maybe version or compilation issue?

KeithWiles commented 4 months ago

Some times wireshark reports checksum wrong because the NIC is doing the checksuming in hardware and it not a real problem. Not sure why in your case.