search

luigirizzo / netmap

Automatically exported from code.google.com/p/netmap
BSD 2-Clause "Simplified" License
1.86k stars 536 forks source link

Netmap cannot see VLAN tags #259

Closed debian89 closed 7 years ago

debian89 commented 7 years ago

Hello,

I am running bridge from netmap apps. If the traffic is untagged there is no problems. But when the packets is tagged I am receive it on second port without vlan tag.

The test staging is:

Sender -> Netmap bridge -> Receiver

Is it there some solution for resolving of this problem?

Thanks!

vmaffione commented 7 years ago

Hi, Interesting point. What NIC are you using?

I was able to reproduce this problem using a simpler setup: a QEMU VM with an e1000 emulated NIC, which receives VLAN tagged ethernet frames. However, this issue is not related to netmap nor to the bridge application. Netmap does not care about what is inside the ethernet packet (except for VALE, but this is not the case), so the problem cannot be there.

After a while I figured out that the e1000 NIC strips the VLAN id from the packet and put it into a field of the RX descriptor. This feature is called RX VLAN offloading and it is basically an hardware acceleration technique. This is why netmap cannot see the VLAN tag in the packet. To let netmap see the VLAN tag you need to disable this offloading (together with the usual ones). This worked for me:

   # ethtool -K eth0 rxvlan off     # in addition to disabling tso, gro, rx, tx, etc.
gl000008 commented 7 years ago

Hi,

I have tried disabling the RX VLAN offloading with the above command (together with the rest of the offloading settings: tx off rx off gso off tso off gro off lro off) and the netmap program still does not capture the original packet with the VLAN header.

In this setup I am using 2 VirtualBox VMs (host1 and host2) that are connected together through another VM with netmap and 2 e1000 NICs (kernel 4.10.13). Both host1 and host2 use VLAN 11 on their interfaces, and the "bridge" application forwards the traffic between them. A TCPDUMP capture at host1 shows that the VLAN header is sent (8100 000b):

host1# tcpdump -enxx -i enp0s16
tcpdump: WARNING: enp0s16: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s16, link-type EN10MB (Ethernet), capture size 65535 bytes
PING 10.1.7.92 (10.1.7.92) from 10.1.7.91 : 56(84) bytes of data.
20:17:25.801838 08:00:27:d5:75:da > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 11, p 0, ethertype ARP, Request who-has 10.1.7.92 tell 10.1.7.91, length 28
        0x0000:  ffff ffff ffff 0800 27d5 75da 8100 000b
        0x0010:  0806 0001 0800 0604 0001 0800 27d5 75da
        0x0020:  0a01 075b 0000 0000 0000 0a01 075c

However, after "bridge" forwards the packet, host2 receives it without the VLAN header (because that is how it was captured):

host2# tcpdump -enxx -i enp0s3
tcpdump: WARNING: enp0s3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), capture size 65535 bytes
20:17:25.802342 08:00:27:d5:75:da > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.1.7.92 tell 10.1.7.91, length 46
        0x0000:  ffff ffff ffff 0800 27d5 75da 0806 0001
        0x0010:  0800 0604 0001 0800 27d5 75da 0a01 075b
        0x0020:  0000 0000 0000 0a01 075c 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000

I have tried ethtool with rxvlan on and off and in both cases the VLAN header is removed. Any ideas how to capture the original packets?

Thanks!

vmaffione commented 7 years ago

Netmap is completely unaware of VLANs. The behaviour depends only on how you configured the netmap VM. For sure you need to disable vlan offloadings (rx/tx) on both interfaces in the netmap VM. Did you create vlan interfaces in the netmap VM? Which netmap interfaces is the bridge program using? Are you using the native netmap adapter?

gl000008 commented 7 years ago

I did not create VLAN interfaces in the netmap VM. The bridge program is using 2 e1000 interfaces (enp0s3 and enp0s16) using the native netmap adapter:

netmap# lsmod |grep netmap
netmap                122880  1 e1000
netmap# ./bridge -i netmap:enp0s3 -i netmap:enp0s16 -v

The offload parameters of the interfaces are the following:

netmap# ethtool -k enp0s3                  
Features for enp0s3:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]
netmap# ethtool -k enp0s16
Features for enp0s16:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]

By the way, just for testing purposes, if I stop the bridge program and run TCPDUMP on the receiving interface of the netmap system, if seems like with RX VLAN acceleration disabled TCPDUMP does not get the VLAN header (note that I have manually set the MAC addresses at host1 and host2 to prevent ARP messages):

netmap# ethtool -K enp0s16  rxvlan off txvlan off
Cannot change tx-vlan-offload
Actual changes:
rx-vlan-offload: off
tx-vlan-offload: off [fixed]
netmap# tcpdump -nxxe -i enp0s16
tcpdump: WARNING: enp0s16: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s16, link-type EN10MB (Ethernet), capture size 65535 bytes
02:57:37.175651 08:00:27:d5:75:da > 08:00:27:c6:90:5c, ethertype IPv4 (0x0800), length 102: 10.1.7.91 > 10.1.7.92: ICMP echo request, id 4818, seq 2, length 64
        0x0000:  0800 27c6 905c 0800 27d5 75da 0800 4500
        0x0010:  0054 af7a 4000 4001 6876 0a01 075b 0a01
        0x0020:  075c 0800 c950 12d2 0002 5f4e ab59 0000
        0x0030:  0000 4760 0b00 0000 0000 1011 1213 1415
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
        0x0060:  3637 13f0 c3df

however, with RX VLAN acceleration enabled, TCPDUMP gets the VLAN header:

netmap# ethtool -K enp0s16  rxvlan on txvlan on
Cannot change tx-vlan-offload
Actual changes:
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
netmap# tcpdump -nxxe -i enp0s16
tcpdump: WARNING: enp0s16: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s16, link-type EN10MB (Ethernet), capture size 65535 bytes
02:58:57.146371 08:00:27:d5:75:da > 08:00:27:c6:90:5c, ethertype 802.1Q (0x8100), length 102: vlan 11, p 0, ethertype IPv4, 10.1.7.91 > 10.1.7.92: ICMP echo request, id 4867, seq 1, length 64
        0x0000:  0800 27c6 905c 0800 27d5 75da 8100 000b
        0x0010:  0800 4500 0054 fe2d 4000 4001 19c3 0a01
        0x0020:  075b 0a01 075c 0800 383f 1303 0001 d153
        0x0030:  ab59 0000 0000 6f3c 0200 0000 0000 1011
        0x0040:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021
        0x0050:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031
        0x0060:  3233 3435 3637

I do not know if this is the expected behaviour.

vmaffione commented 7 years ago

It's not expected. From your description it seems that Virtual Box VLAN emulation is incorrect with rx-vlan disabled, independently of netmap. This makes sense, as netmap does not touch/see/strip the VLAN header, you just get what is in the e1000 ring. Try using QEMU instead for VirtualBox.

gl000008 commented 7 years ago

I tried the same setup with real hardware and ixgbe NICs and it works correctly with VLAN offloading disabled, as you indicated (I will try with QEMU sometime to confirm that the problem was the combination of VirtualBox and e1000). @vmaffione thank you very much for your help!

ramaza commented 7 years ago

I hope it's ok to comment here instead of starting an almost similar thread.

When using QEMU in combination with Virtio-Net driver then Netmap application only sees VLAN tags when the patched Virtio-Net driver is used.

The VM with the Netmap application doesn't have any VLAN subinterfaces configured. And turning VLAN-Offloading on or off doesn't change the behavior.

In the meantime I have disabled also all other offloading features as well. Just to make sure. But with the emulated mode the Netmap application still receives the packets with the VLAN tags stripped off.

I'm now trying to understand if this is a known limitation of the emulated Netmap mode. Or if it's just a wrong behavior of the Virtio-Net driver. Of course I could stick with the patched Virtio driver but I thought it's worth to explore this in case others have the same problem.

vmaffione commented 7 years ago

Hi, Yes, it's a limitation of the netmap emulated mode. It does not depend on a specific driver (I was able to reproduce it with a r8169 and e1000 drivers), and it is not affected by rx-vlan-offload being active or not.

The code location where the emulated mode intercepts packets received from the driver is after this point http://elixir.free-electrons.com/linux/latest/source/net/core/dev.c#L4196, where the VLAN tag is stripped if present (and if this was not already done by the hardware, that's why enabling/disabling rx-vlan-offload has no effect), and put in the sk_buff metadata. This means that netmap can only receive the stripped packet.

Good to know that there is this limitation.

vmaffione commented 7 years ago

By the way, although I didn't remember about that, I already found this limitation and documented it https://github.com/luigirizzo/netmap/blob/master/LINUX/README#L267-L268.