RESOLVED [Edge Throughput] L2 networking issue causes data packets to disappear

[lixuna]

Problem:

• Data Packets are disappearing on L2 network between physical machines on packet.net
  • some packets go through, ie. ARP
  • some packets disappear, ie. UDP

Solution Found: Turn on VF trust in host for interfaces. See https://github.com/cncf/cnfs/issues/61#issuecomment-420459167

---

These tests are targeting the public service provider Packet.net on machine types publicly available.

Other labs, cloud providers and private networks could produce different results. ( example: fd.io CSIT lab using Intel x710 NICs works as expected)

Goal: Use high performance traffic from one host to another through VPP vSwitch on second host

• Accelerated by DPDK

Test configuration setup (goal)

Common host configuration

• Notes for configuring VPP with Mellanox Connectx-4 (PFs and VFs)

---

Tasks:

• [x] re-create interfaces in host on test machines to support iperf

• [x] Test sending data between machines using iperf without DPDK

• [x] Setup DPDK interfaces in host on test machines

• [x] Setup and test with VPP

• [x] document test setup including system configuration

• SKIPPED: send test setup to Packet support team

• SKIPPED: spin up new machines using the new documentation

• SKIPPED: grant access to machines to packet support team

[taylor]

Testing iperf between two machines

Two physical machines running Ubuntu 18.04 LTS on Packet.net. Install iperf software.

Configuration and testing: Pktgen host

• Physical interface attached to CNF host
• Setup virtual interface attached to physical interface
• Setup IP address on virtual interface in host
• Run iperf in client mode sending UDP traffic to CNF host IP

CNF Host

• Physical interface attached to Pktgen host
• Setup virtual interface attached to physical interface
• Setup IP address on virtual interface in host
• Run iperf in server mode

Result:

UDP packets seen on CNF host as expected

Ping and ARP ping

Using ping or arping work as expected

[taylor]

Testing iperf through VPP vSwitch on CNF Host to iperf listening to a tap interface

Configuration: Pktgen

• Physical interface attached to CNF host
• Setup virtual interface attached to physical interface
• Setup 1st IP address on virtual interface in host

CNF Host

• Physical interface attached to Pktgen host
• Setup virtual interface attached to physical interface
• vSwitch - VPP
  • Setup tap interface
  • Bridge tap interface to virtual interface
• Iperf will listen on tap interface

Test:

On Pktgen host, run iperf in client mode on 1st interface sending UDP traffic to CNF host IP On CNF host, run iperf server listening on tap interface (created by VPP)

Create TAP on CNF Host:

ip tuntap add mode tap tap0
ip addr add 172.17.120.20/24 dev tap0
ip link set up tap0

vSwitch (Host VPP) Configuration:

create bridge-domain 1

tap connect tap0 hwaddr random

set int l2 bridge TwentyFiveGigabitEthernet5e/0/4 1
set int l2 bridge tapcli-0 1

set int state TwentyFiveGigabitEthernet5e/0/4 up
set int state tapcli-0 up

Results:

• Initial tests with Ping shows that pings are sent from Tap (CNF Host) to Pktgen interface. Response from Pktgen interface never reaches VPP on other side.

Conclusion: Could indicate that the problem is somewhere in receive side of DPDK when connected to Mellanox VF.

Additional related testing/information:

ARP request from Pktgen host to tap interface on CNF host

• PASSES: w/o VPP running
• PASSES: with VPP running
• Rx, broadcast and unicast are going through as expected

ARP request from CNF host to Pktgen host

• PASSES: w/o VPP running
• FAILS: with VPP running
  • ARP request sent from CNF host as expected
  • ARP request is received on Pktgen host as expected
  • ARP reply is sent from Pktgen host as expected
  • FAIL: ARP reply is never seen on CNF host
    • NIC rx counter increases at 3 packets / second when arping is running

[taylor]

Using a host bridge with virtual interface - no VPP / simple test

Configuration on CNF Host:

ip link add name br0 type bridge
ip link set br0 up
ip link set enp94s0f4 up
ip link set enp94s0f4 master br0
ip addr add 172.16.120.20/24 dev br0

# To see bridge
# bridge link

Testing On Pktgen Host run:

ping -I enp94s0f4 172.16.120.20
arping -i enp94s0f4 172.16.120.20

On CNF Host

ping -I br0 172.16.120.10
arping -i br0 172.16.120.10

Results:

• PASS: ARP requests from Pktgen host to bridge interface on CNF host
• PASS: ARP requests from CNF host to Pktgen host
• PASS: ICMP requests from Pktgen host to bridge interface on CNF host
• PASS: ICMP requests from CNF host to Pktgen host

---

After testing is complete the bridge can be removed with:

ip link set enp94s0f4 nomaster
ip link set enp94s0f4 down
ip lin delete br0 type bridge

[taylor]

Testing with host virtual interface inside of VPP

Using a host virtual interface in VPP on the CNF host

VPP configuration (/etc/vpp/setup.gate) on CNF Host:

create host-interface name enp94s0f4
set interface state host-enp94s0f4 up
set interface ip address host-enp94s0f4 172.16.120.20/24

Testing On Pktgen Host run:

ping -I enp94s0f4 172.16.120.20
arping -i enp94s0f4 172.16.120.20

On CNF Host

ping -I br0 172.16.120.10
arping -i br0 172.16.120.10

Results:

• PASS: ARP requests from Pktgen host to bridge interface on CNF host
  • NOTE: tx not sent from CNF HOST
• FAIL: ARP requests from CNF host to Pktgen host
• FAIL: ICMP requests from Pktgen host to bridge interface on CNF host
• FAIL: ICMP requests from CNF host to Pktgen host

[taylor]

Two solutions:

1. Turn on VF trust for the interfaces. See https://community.mellanox.com/docs/DOC-2473
2. “Spoof” the mac address from the host in VPP (Only for verification. Not a good choice)

Option 1, turn on trust for VF

In the host

ip link set enp94s0f1 vf 1 trust on
ip link set enp94s0f1 vf 0 trust on

Option 2, (Only for verification) spoof mac address:

create host-interface name enp94s0f4 hw-addr <MAC address from enp94s0f4 in host>
set interface state host-enp94s0f4 up
set interface ip address host-enp94s0f4 172.16.120.20/24