Pingpong doesn't ping-pong

wonkyoc commented 3 years ago

Problem

POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: f8 b4 6a a0 7a fc 18 31 bf cf f8 5e 08 00 45 00 ..j.z..1...^..E.
   16: 00 2c 00 00 00 00 40 11 66 bf 0a 00 00 01 0a 00 .,....@.f.......
   32: 00 02 27 11 27 10 00 18 00 00 50 51 e4 00 d7 55 ..'.'.....PQ...U
   48: 00 00 27 67 7b f5 44 7f 00 00 00 00

I got this message at the ping-side so it seems that the pong-side does not send a packet in a right way. As a result, a looping does not work at all. Is it matter in the core of UDPDK? Or should I change something in the application-side?

leoll2 commented 3 years ago

Hello @wonkyoc, could you share more details about your environment? NIC, driver, ... Also, did you configure config.ini according to your system?

wonkyoc commented 3 years ago

Hi @leoll2, Here is my NICs:

# Driver
Ping-side: Intel Corporation Ethernet Connection I219-LM, configured with IGB_UIO
Pong-side: Intel Corporation I210 Gigabit Network Connection, configured with VFIO-PCI

# config.ini - ping-side
[port0]
mac_addr=f8:b4:6a:a0:7a:fc
ip_addr=10.0.0.2

[port0_dst]
mac_addr=18:31:bf:cf:f8:5e

# config.ini - pong-side
[port0]
mac_addr=18:31:bf:cf:f8:5e
ip_addr=10.0.0.1

[port0_dst]
mac_addr=f8:b4:6a:a0:7a:fc

Both devices are attached on the motherboard so they are not separately detached. I've checked that both devices configured with the same driver (IGB_UIO), but it does not wok as well. When the ping-side sends a packet to the pong-side, it seems there is no problem. However, when the pong-side sends a packet to the ping-side for the reply purpose, the ping-side receives the non-IPv4 packet.

I even checked rte_mbuf->packet_type right before it is sent to the flush_tx_table at the pong side. But, for some reason, these information are set as zero value when I check reassemble() function at the ping-side.

So, I guess the problem may not be from the udpdk, but device-side. Any helps would be appreciated. Thanks

Also, did you configure config.ini according to your system?

Yes. I set it on both side correctly. I checked the right ping-side IP address when the pong-side received the payload.

leoll2 commented 3 years ago

Interesting. Before we investigate more into detail, can you try if the pktgen app works (in both directions)?

wonkyoc commented 3 years ago

[Configuration A]
Sender: Intel Corporation Ethernet Connection I219-LM, configured with IGB_UIO
Receiver: Intel Corporation I210 Gigabit Network Connection, configured with IGB_UIO

[Configuration B]
Sender: Intel Corporation I210 Gigabit Network Connection, configured with IGB_UIO
Receiver: Intel Corporation Ethernet Connection I219-LM, configured with IGB_UIO

I tested both configuration and I found that configuration B does not work. It produces this error:

# Configuration A dump before `reassemble()`& works fine
Dumping payload [len = 142]:
    0: 18 31 bf cf f8 5e f8 b4 6a a0 7a fc 08 00 45 00 .1...^..j.z...E.
   16: 00 80 00 00 00 00 40 11 66 6a 0a 00 00 02 0a 00 ......@.fj......
   32: 00 02 27 10 27 11 00 6c 00 00 4c 6f 72 65 6d 20 ..'.'..l..Lorem 
   48: 69 70 73 75 6d 20 64 6f 6c 6f 72 20 73 69 74 20 ipsum dolor sit 
   64: 61 6d 65 74 2c 20 63 6f 6e 73 65 63 74 65 74 75 amet, consectetu
   80: 72 20 61 64 69 70 69 73 63 69 6e 67 20 65 6c 69 r adipiscing eli
   96: 74 2e 20 41 6c 69 71 75 61 6d 20 76 61 72 69 75 t. Aliquam variu
  112: 73 20 73 65 6d 70 65 72 20 66 61 75 63 69 62 75 s semper faucibu
  128: 73 2e 20 56 69 76 61 6d 75 73 20 63 6f 6e 

# Configuration B dump & error
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 142]:
    0: f8 b4 6a a0 7a fc 18 31 bf cf f8 5e 08 00 45 00 ..j.z..1...^..E.
   16: 00 80 00 00 00 00 40 11 66 6b 0a 00 00 01 0a 00 ......@.fk......
   32: 00 02 27 10 27 11 00 6c 00 00 4c 6f 72 65 6d 20 ..'.'..l..Lorem 
   48: 69 70 73 75 6d 20 64 6f 6c 6f 72 20 73 69 74 20 ipsum dolor sit 
   64: 61 6d 65 74 2c 20 63 6f 6e 73 65 63 74 65 74 75 amet, consectetu
   80: 72 20 61 64 69 70 69 73 63 69 6e 67 20 65 6c 69 r adipiscing eli
   96: 74 2e 20 41 6c 69 71 75 61 6d 20 76 61 72 69 75 t. Aliquam variu
  112: 73 20 73 65 6d 70 65 72 20 66 61 75 63 69 62 75 s semper faucibu
  128: 73 2e 20 56 69 76 61 6d 75 73 20 63 6f 6e

~~It seems like there are more values (f8 b4 6a a0 7a fc) in front of (18 31 bf cf f8 5e 08 00 45 00). I am not sure why this is added.~~

wonkyoc commented 3 years ago

~~Oh, I see the problem. It is not a new value. They are just flipped. I am not sure why this problem occurs.~~ I realized that they are MAC addresses.

Configuration A: 18 31 bf cf f8 5e    f8 b4 6a a0 7a fc
Configuration B: f8 b4 6a a0 7a fc    18 31 bf cf f8 5e

leoll2 commented 3 years ago

Yes, what you are looking at are the full packets, including the headers. If you swap the sender and the receiver, then also their MAC address will be swapped in the packet indeed.

You can analyze the content of the packets by means of one of many decoders available online, e.g. https://hpd.gasmi.net/.

In your case, the full hex packets are:

18 31 BF CF F8 5E F8 B4 6A A0 7A FC 08 00 45 00 00 80 00 00 00 00 40 11 66 6A 0A 00 00 02 0A 00 00 02 27 10 27 11 00 6C 00 00 4C 6F 72 65 6D 20 69 70 73 75 6D 20 64 6F 6C 6F 72 20 73 69 74 20 61 6D 65 74 2C 20 63 6F 6E 73 65 63 74 65 74 75 72 20 61 64 69 70 69 73 63 69 6E 67 20 65 6C 69 74 2E 20 41 6C 69 71 75 61 6D 20 76 61 72 69 75 73 20 73 65 6D 70 65 72 20 66 61 75 63 69 62 75 73 2E 20 56 69 76 61 6D 75 73 20 63 6F 6E

and

F8 B4 6A A0 7A FC 18 31 BF CF F8 5E 08 00 45 00 00 80 00 00 00 00 40 11 66 6B 0A 00 00 01 0A 00 00 02 27 10 27 11 00 6C 00 00 4C 6F 72 65 6D 20 69 70 73 75 6D 20 64 6F 6C 6F 72 20 73 69 74 20 61 6D 65 74 2C 20 63 6F 6E 73 65 63 74 65 74 75 72 20 61 64 69 70 69 73 63 69 6E 67 20 65 6C 69 74 2E 20 41 6C 69 71 75 61 6D 20 76 61 72 69 75 73 20 73 65 6D 70 65 72 20 66 61 75 63 69 62 75 73 2E 20 56 69 76 61 6D 75 73 20 63 6F 6E

Both are IPv4 UDP packets, so yeah the error can be a bit misleading. In the former packet, I notice that the source and destination IP are identical, which is odd and hints at an improper configuration. Anyway, the error that you see is produced in udpdk_poller.c:363 when the header does not pass the RTE_ETH_IS_IPV4_HDR check; the latter is a simple macro defined by DPDK as #define RTE_ETH_IS_IPV4_HDR(ptype) ((ptype) & 0x00000010). If you're confident to exclude a configuration issue, then I'd suggest to debug this section of the code to find out why the check fails.

Let me know if you make progress. Good luck!

wonkyoc commented 3 years ago

Thanks for your link! The one that I provided was misconfigured, but I don't think it is directly related to this issue. For more detail, I used rte_pktmbuf_dump() and I found ptype is not correctly sent.

Dump

Sender

// udpdk_poller.c: 526
...
struct rte_mbuf *m = tx_mbuf_table[0];
rte_pktmbuf_dump(stdout, m, 256);
flush_tx_table(tx_mbuf_table, tx_count);
...

Receiver

// udpdk_poller.c: 332
...
rte_pktmbuf_dump(stdout, m, 256);
eth_hdr = rte_pktmbuf_mtod(m, struct rte_ether_hdr *);
...

Result

Sender

dump mbuf at 0x10138c640, iova=0x211f8c6c0, buf_len=2176
  pkt_len=142, ol_flags=0, nb_segs=1, port=65535, ptype=0x211
  segment at 0x10138c640, data=0x10138c740, len=142, off=128, refcnt=1
  Dump data at [0x10138c740], len=142
F8 B4 6A A0 7A FC 18 31 BF CF F8 5E 08 00 45 00 
0 80 00 00 00 00 40 11 66 6B 0A 00 00 01 0A 00
00 02 27 10 27 11 00 6C 00 00 4C 6F 72 65 6D 20 
69 70 73 75 6D 20 64 6F 6C 6F 72 20 73 69 74 20 
61 6D 65 74 2C 20 63 6F 6E 73 65 63 74 65 74 75 
72 20 61 64 69 70 69 73 63 69 6E 67 20 65 6C 69 
74 2E 20 41 6C 69 71 75 61 6D 20 76 61 72 69 75
73 20 73 65 6D 70 65 72 20 66 61 75 63 69 62 75
73 2E 20 56 69 76 61 6D 75 73 20 63 6F 6E

Receiver

# Receiver: the packet was dumped before `reassemble()`
dump mbuf at 0x10058c640, iova=0x1d9b8c6c0, buf_len=2176                                                                                                                               
  pkt_len=142, ol_flags=0, nb_segs=1, port=0, ptype=0                                                                                                                                  
  segment at 0x10058c640, data=0x10058c740, len=142, off=128, refcnt=1                                                                                                                 
  Dump data at [0x10058c740], len=142                                                                                                                                                  
F8 B4 6A A0 7A FC 18 31 BF CF F8 5E 08 00 45 00
00 80 00 00 00 00 40 11 66 6B 0A 00 00 01 0A 00
00 02 27 10 27 11 00 6C 00 00 4C 6F 72 65 6D 20
69 70 73 75 6D 20 64 6F 6C 6F 72 20 73 69 74 20
61 6D 65 74 2C 20 63 6F 6E 73 65 63 74 65 74 75
72 20 61 64 69 70 69 73 63 69 6E 67 20 65 6C 69
74 2E 20 41 6C 69 71 75 61 6D 20 76 61 72 69 75
73 20 73 65 6D 70 65 72 20 66 61 75 63 69 62 75
73 2E 20 56 69 76 61 6D 75 73 20 63 6F 6E

I am not sure why ptype cannot be correctly sent although other variables are sent correctly. I still don't understand how rte_mbuf metadata (e.g. ptype or pkt_len) is sent because wireshark cannot capture its metadata, but can get the UDP data. I guess the problem is on the device side or maybe on the dpdk itself. I will let you know when I find the root cause.

wonkyoc commented 3 years ago

I have not found the root cause, but I solved this issue by using another device. So, I am closing this.

leoll2 / UDPDK