Performance: Throughput with OVS drops from 9.9Gbps to 6Gbps with VXLAN tunneling

[radhikaniranjan]

The throughput with VXLAN tunneling drops to 6 Gbps:

Client connecting to 10.0.85.3, TCP port 5001

TCP window size: 96.7 KByte (default) [ 3] local 10.0.101.3 port 51781 connected with 10.0.85.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 7.46 GBytes 6.41 Gbits/sec

I am using an MTU of 9000. Without vxlan, the same ovsbridge gives 9.89 Gbps.

Here are other setup details: My set up is as follows: OVS (10.0.85.3) <-----VXLAN ----PHYLINK (15.0.85.3) ===============PHYLINK(15.0.101.2)---VXLAN-----> OVS(10.0.101.2) PHYLINK is my 10Gbps physical link.

Here is the ovs configuration

Host 1: ovs-vsctl show 9399da11-e2ae-49d5-b5c8-08c6864ad7ab Bridge ovsbr Port "vx1" Interface "vx1" type: vxlan options: {remote_ip="15.0.101.3"} Port ovsbr Interface ovsbr

type: internal Host 2: ovs-vsctl show 94389297-4998-4960-b120-a83f4f2cc4d1 Bridge ovsbr Port "vx1" Interface "vx1" type: vxlan options: {remote_ip="15.0.85.3"} Port ovsbr Interface ovsbr

type: internal I have a lot of interfaces, therefore showing only the relevant ones:

Host 1:

ovsbr: flags=4163 mtu 9000 inet 10.0.85.3 netmask 255.255.0.0 broadcast 10.0.255.255 inet6 fe80::f0f4:67ff:fe89:2348 prefixlen 64 scopeid 0x20 ether f2:f4:67:89:23:48 txqueuelen 0 (Ethernet) RX packets 6076890 bytes 51004101244 (47.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1941275 bytes 6326190860 (5.8 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

p1p1: flags=4163 mtu 9000 inet 15.0.85.3 netmask 255.255.0.0 broadcast 15.0.255.255 inet6 fe80::92e2:baff:fe26:82d4 prefixlen 64 scopeid 0x20 ether 90:e2:ba:26:82:d4 txqueuelen 1000 (Ethernet) RX packets 5584777 bytes 49719271982 (46.3 GiB) RX errors 0 dropped 906 overruns 0 frame 0 TX packets 1729674 bytes 200678713 (191.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Host 2: ovsbr Link encap:Ethernet HWaddr 16:03:b5:06:56:40 inet addr:10.0.101.3 Bcast:10.0.255.255 Mask:255.255.0.0 inet6 addr: fe80::260:ddff:fe46:515f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:8727829 errors:0 dropped:12 overruns:0 frame:0 TX packets:7412586 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:14258252105 (14.2 GB) TX bytes:156935144728 (156.9 GB)

eth5 Link encap:Ethernet HWaddr 00:60:dd:46:51:5f inet addr:15.0.101.3 Bcast:15.0.255.255 Mask:255.255.0.0 inet6 addr: fe80::260:ddff:fe46:515f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:56992363 errors:0 dropped:0 overruns:0 frame:0 TX packets:69611348 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:17908270611 (17.9 GB) TX bytes:526649690661 (526.6 GB) Interrupt:77

[mestery]

This should be fixed with my last commit, can you try it and verify? In my setup, VXLAN actually performs better than GRE even.

[radhikaniranjan]

Thanks! I'll check and let you know. Was there a specific issue that caused the throughput drop? I was just curious. I do understand if the problem is too involved to explain or was a result of multiple small things.

[radhikaniranjan]

Hi Kyle,

Thanks for the update. I tried the latest branch, but I don’t see an improvement. Perhaps there is some config parameter that I am missing? I’d love to know insights on what are the knobs I could use to change the perf that I am seeing.

Here are my testbed details: physical links (both ends have 9000 MTU): 15.0.113.3 and 15.0.101.3 openvswitch with vxlan(both ends have 9000 MTU): 11.0.113.3 and 11.0.101.3

RX, Baseline (on physical link): 9.9Gbps RX, VXLAN(openvswitch): 5.42 Gbps

TX, Baseline (on physical link): 8.32Gbps TX, VXLAN(openvswitch): 4.57 Gbps

Output of ovs-ofctl show, ovs-dpctl show, ovs-vsctl show: Server 1) ovs-ofctl show ovsbr OFPT_FEATURES_REPLY (xid=0×1): dpid:00009e713aab6145 n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE 1(vx1): addr:5a:30:46:d7:73:b1 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(ovsbr): addr:9e:71:3a:ab:61:45 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0×3): frags=normal miss_send_len=0

ovs-dpctl show system@ovs-system: lookups: hit:3044610 missed:30 lost:0 flows: 0 port 0: ovs-system (internal) port 1: ovsbr (internal) port 2: vx1 (vxlan: remote_ip=15.0.101.3)

ovs-vsctl show b7081fed-9be3-4243-893e-94b4b50211d8 Bridge ovsbr Port ovsbr Interface ovsbr type: internal Port “vx1″ Interface “vx1″ type: vxlan options: {remote_ip=”15.0.101.3″}

Server 2) ovs-ofctl show ovsbr OFPT_FEATURES_REPLY (xid=0×1): dpid:00001eb955408244 n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE 1(vx1): addr:5a:b1:02:dd:b7:15 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(ovsbr): addr:1e:b9:55:40:82:44 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0×3): frags=normal miss_send_len=0

ovs-dpctl show system@ovs-system: lookups: hit:2794001 missed:30 lost:0 flows: 0 port 0: ovs-system (internal) port 1: ovsbr (internal) port 2: vx1 (vxlan: remote_ip=15.0.113.3)

ovs-vsctl show 6a1f48dd-30d8-4d1b-b52a-4698354e5c26 Bridge ovsbr Port “vx1″ Interface “vx1″ type: vxlan options: {remote_ip=”15.0.113.3″} Port ovsbr Interface ovsbr type: internal

Here’s the modinfo for the openvswitch module: Server 1) modinfo openvswitch filename: /lib/modules/3.5.0-030500-generic/kernel/net/openvswitch/openvswitch.ko version: 1.9.90 license: GPL description: Open vSwitch switching datapath srcversion: 726FDD22BBD59C95CB6769A depends: vermagic: 3.5.0-030500-generic SMP mod_unload modversions

Server 2) modinfo openvswitch filename: /lib/modules/3.5.0-17-generic/kernel/net/openvswitch/openvswitch.ko version: 1.9.90 license: GPL description: Open vSwitch switching datapath srcversion: 726FDD22BBD59C95CB6769A depends: vermagic: 3.5.0-17-generic SMP mod_unload modversions

Please let me know if there is any other debug info I can provide.

Thanks Radhika

[mestery]

Radhika:

Can you do me a favor: Please baseline this test against GRE tunnels as well. In my testing, I was comparing performance results against GRE tunnels, and in fact the VXLAN tunnels seem to run slightly faster than the GRE tunnels. I just compared against non tunneled traffic, and the drop-off in tunneling is pretty significant. I'll keep digging into that now. But if you could verify that GRE tunnels perform the same as VXLAN, at least that particular performance drop-off can be marked as addressed.

Thanks, Kyle

[radhikaniranjan]

Thanks Kyle, I'll get back to you with the tests but I wanted to confirm that in all my previous testing of GRE vs VXLAN I have always found GRE tunnels to have poorer throughput than VXLAN tunnels. HTH.