mwiget
commented
5 years ago First attempt turned out to be easier than expected: https://github.com/mwiget/snabb/tree/passthru
Running lwaftr on-a-stick over 10GE loopback physical link to packetblaster (enhanced with passthru as well), while running iperf between both tap interfaces works.
Configured tap xe0 interface in the network namespace lwaftr, where snabb lwaftr will be run is configured with an IP address. Its MAC address is configured in the passthru-interface section. Note: This can also be a MAC address of a connecting device behind xe0, which is closer to the snabbvmx use case:
$ sudo ip netns exec lwaftr ifconfig
xe0: flags=4099<UP,BROADCAST,MULTICAST> mtu 9000
inet 10.9.9.2 netmask 255.255.255.0 broadcast 10.9.9.255
inet6 fe80::10fe:11ff:fef1:b41d prefixlen 64 scopeid 0x20<link>
ether 12:fe:11:f1:b4:1d txqueuelen 1000 (Ethernet)
RX packets 50 bytes 5606 (5.6 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 33 bytes 2462 (2.4 KB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0The relevant snabb lwaftr configuration section:
instance {
device "03:00.0";
queue {
id 0;
external-interface {
ip 172.20.0.100;
mac 02:22:22:22:22:22;
next-hop {
mac 04:44:44:44:44:44;
}
}
internal-interface {
ip 2001:db8::100;
mac 02:22:22:22:22:22;
next-hop {
mac 04:44:44:44:44:44;
}
}
passthru-interface {
device xe0;
mac 12:fe:11:f1:b4:1d;
mtu 9000;
}
}
}Pinging thru the interface while serving 3MPPS IMIX traffic:
$ sudo ip netns exec lwaftr tcpdump -n -e -i xe0
16:44:55.387944 12:fe:11:f1:b4:1d > 42:18:e3:a8:fa:80, ethertype ARP (0x0806), length 42: Request who-has 10.9.9.1 tell 10.9.9.2, length 28
16:44:55.388153 42:18:e3:a8:fa:80 > 12:fe:11:f1:b4:1d, ethertype ARP (0x0806), length 60: Request who-has 10.9.9.2 tell 10.9.9.1, length 46
16:44:55.388160 12:fe:11:f1:b4:1d > 42:18:e3:a8:fa:80, ethertype ARP (0x0806), length 42: Reply 10.9.9.2 is-at 12:fe:11:f1:b4:1d, length 28
16:44:55.388708 42:18:e3:a8:fa:80 > 12:fe:11:f1:b4:1d, ethertype ARP (0x0806), length 60: Reply 10.9.9.1 is-at 42:18:e3:a8:fa:80, length 46
16:44:56.316105 42:18:e3:a8:fa:80 > 12:fe:11:f1:b4:1d, ethertype IPv4 (0x0800), length 98: 10.9.9.1 > 10.9.9.2: ICMP echo request, id 1278, seq 7, length 64
16:44:56.316121 12:fe:11:f1:b4:1d > 42:18:e3:a8:fa:80, ethertype IPv4 (0x0800), length 98: 10.9.9.2 > 10.9.9.1: ICMP echo reply, id 1278, seq 7, length 64
16:44:57.340088 42:18:e3:a8:fa:80 > 12:fe:11:f1:b4:1d, ethertype IPv4 (0x0800), length 98: 10.9.9.1 > 10.9.9.2: ICMP echo request, id 1278, seq 8, length 64
16:44:57.340104 12:fe:11:f1:b4:1d > 42:18:e3:a8:fa:80, ethertype IPv4 (0x0800), length 98: 10.9.9.2 > 10.9.9.1: ICMP echo reply, id 1278, seq 8, length 64v6+v4: 1.500+1.500 = 2.999946 MPPS, 4.462+3.982 = 8.443490 Gbps, lost 0.000%
v6+v4: 1.500+1.500 = 2.999663 MPPS, 4.461+3.981 = 8.442705 Gbps, lost 0.005%
v6+v4: 1.500+1.500 = 2.999976 MPPS, 4.462+3.982 = 8.443567 Gbps, lost 0.000%
v6+v4: 1.500+1.500 = 3.000023 MPPS, 4.462+3.982 = 8.443695 Gbps, lost 0.000%
v6+v4: 1.500+1.500 = 3.000009 MPPS, 4.462+3.982 = 8.443676 Gbps, lost 0.000%
v6+v4: 1.500+1.500 = 2.999997 MPPS, 4.462+3.982 = 8.443622 Gbps, lost 0.000%Script to create the network namespace and Tap interfaces:
$ cat bench.sh
#!/bin/bash
NS=lwaftr
if [ -z "$(ip netns list|grep $NS)" ]; then
echo "creating netns $NS ..."
sudo ip netns add $NS
fi
ip netns list
echo "creating xe0 in netns $NS and default ..."
sudo ip netns exec $NS ip tuntap add dev xe0 mode tap
sudo ip netns exec $NS ifconfig xe0 mtu 9000 10.9.9.2/24 up
sudo ip tuntap add dev xe0 mode tap
sudo ifconfig xe0 mtu 9000 10.9.9.1/24 up
export MACPB=$(ip link show xe0|grep ether|awk '{print $2}')
export MACLW=$(sudo ip netns exec $NS ip link show xe0|grep ether|awk '{print $2}')
echo macpb=$MACPB maclw=$MACLW
envsubst < lwaftr.conf.template > lwaftr1.confThe last command replaces the variable $MACLW in the lwaftr config template with the actual MAC address of the xe0 interface within the namespace.
To run lwaftr:
$ cat run-lwaftr1.sh
#!/bin/bash
NS=lwaftr
echo "launching iperf server in netns $NS"
sudo ip netns exec $NS iperf -s &
echo "launching lwaftr in netns $NS ..."
sudo ip netns exec $NS snabb/src/snabb lwaftr run --conf lwaftr1.conf -n lwaftr1To run packetblaster:
$ cat run-packetblaster.sh
#!/bin/bash
MACPB=$(ip link show xe0|grep ether|awk '{print $2}')
sudo snabb/src/snabb packetblaster lwaftr \
--src_mac 04:44:44:44:44:44 \
--dst_mac 02:22:22:22:22:22 \
--pci 04:00.0 \
--pass_tap xe0 \
--pass_mac $MACPB \
--ipv4 172.20.9.100 \
--b4 2001:db8::100,193.5.1.100,1024 \
--aftr fc00::100 \
--count 63000 \
--rate 3 \Once lwaftr and packetblaster are launched, one can ping xe0 behind lwaftr from the host:
ping 10.9.9.2
PING 10.9.9.2 (10.9.9.2) 56(84) bytes of data.
64 bytes from 10.9.9.2: icmp_seq=1 ttl=64 time=0.442 ms
64 bytes from 10.9.9.2: icmp_seq=2 ttl=64 time=0.213 ms
^C
--- 10.9.9.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.213/0.327/0.442/0.115 ms
I'm sharing this brain dump to collect feedback while working on a prototype.
Snabbvmx offered a unique way to pass control traffic to an attached VM (e.g. Juniper vMX) to handle all network control traffic, from ARP, IPv6 NDP, BFD, LLDP and routing protocols like ISIS, OSPF and BGP. Snabbvmx also offered next-hop resolution for processed packets by periodically sending a copy of the packet to the VM via the VhostUser interface. vmx-docker-lwaftr uses snabbvmx.
lwAFTR evolved dramatically since snabbvmx was introduced. A YANG file is now configuring more than one instances serving network interfaces and more and more basic network functions like ARP and ICMP have been implemented. This allows for alternate deployment options, where multiple interfaces can be served as slaves "on a stick" from a master snabb instance. Availability and routing of traffic towards each snabb instance can be handled via an "out-of-band" BGP session between the connecting L2/L3 switch and a route server, e.g. ExaBGP or Junos RR. (See Video of crr-snabb-lwaftr prototype).
But this design requires an additional interface for the BGP session between switch and the BGP process and something must monitor the health of each lwAFTR instance and update the routing table. Snabbvmx solved this by "fate sharing" of date and control plane thru each interface.
This brings me to the main idea: Pass control traffic (like BGP) thru one or more interfaces of lwAFTR to an auxiliary TAP interface. LwAFTR runs in hairpinning mode.
These auxiliary interfaces are passing thru traffic between the L2/L3 switch and an attached route reflector. The L2/L3 switch, lwAFTR and the route reflector all have unique IPv4 and IPv6 addresses and MAC addresses per link. IMHO LwAFTR doesn't need to know the route reflectors IP address.
Discussion: It is probably sufficient to know its MAC address. Though it is tempting to ignore the MAC address by simply passing BUM (Broadcast & Unknown Multicast) traffic thru, together with packets on the same subnet. But that would make it impossible to use loopback IP addresses. I'll most likely stick to telling lwAFTR the MAC address of the connected route reflector via configuration.
Configuration
The YANG software-config has already containers for internal-interface and external-interface. This shall be extended with a passthru-interface container. The following example illustrates the idea:
There is no need to specify an IP address for the passhtru-interface; forwarding decisions are made at L2 based on the MAC address and optional VLAN towards the TAP interface. The optional leaf's promiscuous and sample-rate are an idea to use the same tap interface for monitoring purposes and merits further investigation.
Next-hop resolution: snabbvmx uses the nh_fwd app to periodically pass a copy of an egress packet to the connected vMX VM. IMHO it is sufficient to specify the IPv4 and IPv6 next-hop address in the lwAFTR config and have the route reflector interact with Snabb config to learn about the active binding table entries and combine them with the internal- and external-interface IP addresses to create routing entries. By having multiple instances, each connected to a separate interface of the L2/L3 switch offers redundancy.