Tunnel port not processing incoming packets when skb is marked in 2.5.x versions

[jllorente]

Using Ubuntu 16.04 and any version of the OVS 2.5.x, I think I may have come across a bug. I have used iptables to set the skb.mark field using the MARK action for traffic classification in the mangle table. After setting a mark value on the incoming tunneled traffic (VXLAN/GRE), the switch stops processing these packets altogether. If I remove the mark, the packets are processed normally. I have verified this by pinging 2 hosts connected through a VXLAN and GRE tunnel. However, the issue does not seem to be present in any of the OVS 2.7.x versions.

[jllorente]

This test was performed in OvS 2.5.0 but the issue still exists in recent 2.5.3.

Here are the steps to reproduce:

Output of “ovs-vsctl show” f0d81ccd-5b79-4832-ab18-0f874200dda8 Bridge "br1" Port "br1" Interface "br1" type: internal Port "test1s" Interface "test1s" Port "tun1" Interface "tun1" type: vxlan options: {key=flow, local_ip="127.0.0.2", remote_ip="127.0.0.1"} Bridge "br0" Port "br0" Interface "br0" type: internal Port "tun0" Interface "tun0" type: vxlan options: {key=flow, local_ip="127.0.0.1", remote_ip="127.0.0.2"} Port "test0s" Interface "test0s" ovs_version: "2.5.0"

Create supporting virtual network setup

sudo ip netns add ns0 sudo ip netns add ns1

sudo ip link add test0 type veth peer name test0s sudo ip link add test1 type veth peer name test1s

sudo ip link set dev test0s up sudo ip link set dev test1s up

sudo ip link set dev test0 netns ns0 sudo ip link set dev test1 netns ns1

sudo ip netns exec ns0 ip link set dev lo up sudo ip netns exec ns0 ip link set dev test0 up sudo ip netns exec ns0 ip address add dev test0 1.1.1.1/24

sudo ip netns exec ns1 ip link set dev lo up sudo ip netns exec ns1 ip link set dev test1 up sudo ip netns exec ns1 ip address add dev test1 1.1.1.2/24

Create OVS bridges

sudo ovs-vsctl add-br br0 sudo ovs-vsctl add-br br1 sudo ovs-vsctl add-port br0 test0s sudo ovs-vsctl add-port br1 test1s

Create 2 OVS tunnel ports, and assign OpenFlow port 100

sudo ovs-vsctl add-port br0 tun0 -- set interface tun0 ofport_request=100 -- set interface tun0 type=vxlan options:key=flow options:remote_ip=127.0.0.2 options:local_ip=127.0.0.1 sudo ovs-vsctl add-port br1 tun1 -- set interface tun1 ofport_request=100 -- set interface tun1 type=vxlan options:key=flow options:remote_ip=127.0.0.1 options:local_ip=127.0.0.2

Initiate ping between network namespaces

sudo ip netns exec ns0 ping 1.1.1.2

Add iptables marking rule -> ping stops working

sudo iptables -t mangle -A INPUT -d 127.0.0.2 -p udp --dport 4789 -j MARK --set-mark 0xabcdef

Remove iptables marking rule -> ping starts working

sudo iptables -t mangle -D INPUT -d 127.0.0.2 -p udp --dport 4789 -j MARK --set-mark 0xabcdef

[jllorente]

Attending to the instructions received in the mailing list, I have managed to find the culprit via git bisect. The bug I am experiencing seems to be fixed after removing the OVS GRE IPsec tunnel functionality.

Here is the log of the process:

$ git bisect log
git bisect start
# bad: [c298ef781c2d35d939fe163cbc2f41ea7b1cb8d1] Set release date for 2.7.0.
git bisect bad c298ef781c2d35d939fe163cbc2f41ea7b1cb8d1
# good: [f4b0e64cffb4777ff03d48621c3eadcf1d8c19f3] Set release date for 2.6.1.
git bisect good f4b0e64cffb4777ff03d48621c3eadcf1d8c19f3
# good: [4c714486187dada924f1e1fc2fdbec99cddd777f] Prepare for 2.6.0.
git bisect good 4c714486187dada924f1e1fc2fdbec99cddd777f
# bad: [048318e0726c94876e504de7b578f929e9f509a0] netdev: Count ports within mutex.
git bisect bad 048318e0726c94876e504de7b578f929e9f509a0
# bad: [b614c894ee55b07009f3e5b10ddec7556773fcc4] netdev-dpdk: Configure flow control only when necessary.
git bisect bad b614c894ee55b07009f3e5b10ddec7556773fcc4
# good: [cc4583aadd2a55b340bc7917305b41bbba29c9f2] ovn-northd: Add load-balancers to gateway routers.
git bisect good cc4583aadd2a55b340bc7917305b41bbba29c9f2
# good: [9ef3a410b2d93ae6890c6e831dd3ed84d3c374d9] datapath-windows: Add define for last module number
git bisect good 9ef3a410b2d93ae6890c6e831dd3ed84d3c374d9
# good: [9e9d0384910e8d1e96463be19378de3fe9d64877] openvswitch: deprecates support for IPsec tunnel port.
git bisect good 9e9d0384910e8d1e96463be19378de3fe9d64877
# bad: [dd0dc9eda0e0b4a6b2f8f4dee442be6865e60c89] revalidator: Reuse xlate_ukey from deletion.
git bisect bad dd0dc9eda0e0b4a6b2f8f4dee442be6865e60c89
# bad: [9f02d70c11e14366f2136caaedcebd629551a131] ofp-actions: Style fixes.
git bisect bad 9f02d70c11e14366f2136caaedcebd629551a131
# good: [5e8bc3c549ca9bfa02c5525c02cb4ee12ef1f06e] ovsdb: Fix memory leak when disposing 'replication_dbs'
git bisect good 5e8bc3c549ca9bfa02c5525c02cb4ee12ef1f06e
# bad: [9120cfc05cd49d4ba1a47eb97e6407e72a5d33f7] netdev-linux: Use ethtool when miimon fails.
git bisect bad 9120cfc05cd49d4ba1a47eb97e6407e72a5d33f7
# bad: [2b02d770c4cb381ec32cd4b7b1e991c42b448884] openvswitch: Allow external IPsec tunnel management.
git bisect bad 2b02d770c4cb381ec32cd4b7b1e991c42b448884
# first bad commit: [2b02d770c4cb381ec32cd4b7b1e991c42b448884] openvswitch: Allow external IPsec tunnel management.

The details of the commit

2b02d770c4cb381ec32cd4b7b1e991c42b448884 is the first bad commit
commit 2b02d770c4cb381ec32cd4b7b1e991c42b448884
Author: Pravin B Shelar <pshelar@ovn.org>
Date:   Sat Sep 24 11:44:53 2016 -0700

    openvswitch: Allow external IPsec tunnel management.

    OVS GRE IPsec tunnel support has multiple issues, Therefore
    it was deprecated in OVS 2.6.

    Following patch removes support for GRE IPsec and allows external
    IPsec tunnel management for any type of tunnel not just GRE.
    e.g. user can encrypt Geneve or VxLan traffic.

    It can be done by using openflow pipeline to set skb-mark
    and using IPsec keying daemons to implement IPsec tunnels.
    This packet can be matched for the skb-mark to encrypt
    selective tunnel traffic.

    VMware-BZ: 1710701
    Signed-off-by: Pravin B Shelar <pshelar@ovn.org>
    Acked-by: Ansis Atteka <aatteka@ovn.org>

:100644 100644 6e284aac32557ba75408375d58546ac7745433dd 069ab424c644ef0ee992340f450e1dae5fc42454 M      NEWS
:100644 100644 cf5343714937c28b90ca5f3457d23f0ed5df1ebf 53b0fafe3a56e604c2b62097c54ef6dcd135fe83 M      README.md
:040000 040000 636109402f0a3a4798d68f9cf85399b7230d7819 ed14940340a22190843c9270740039e404521b59 M      debian
:040000 040000 5444eb414d2383ec4340243cbfcfcdb0da80e144 2c5687e8813086c083ef2df7382a6fc17f782040 M      lib
:040000 040000 d6cd119e1715c6716dd8fc509169d1499d3a2e31 310aa4ce502847c2824d4b04bd539e60b2100b26 M      ofproto
:040000 040000 208fe072c0f3a3f8139f695019c81fcffcfbd373 6f73eb237766d6bab68fc36389c2ed5fd57e2240 M      tests
:040000 040000 0c796059e07e889bded911f39324d6e9e3cb6771 e5ac1d4e6d492899d2194bf451d5393513e5fea1 M      utilities
:040000 040000 5aa99398dbba68c93985189042fa126bf0c527ed aa9d605b55ac9e79524b9151d4173eaccb8eb9d0 M      vswitchd

Is it feasible to port this to next release of OvS 2.5.3 ?