iperf3 rate missmatch

[jshen28]

Hello,

In a test where iperf3 is used to test virtual network performance, I found following behavior. Where sender sends packet with a rate near 110 thousand while receiver only got 80 thousand, the virtual machine booted with vxlan , and the network topology is

|vm|
 |
|--------|
| br-int |
|--------|
    |
|---------|
|  br-tun |
|---------|

and vxlan port is an internal create on another ovs bridge.

So my question is does openvswitch kernel module internally caches packet and processes it later?

Log:

# iperf3 -c 192.168.0.30 -u -b 0 -l 16 -V --get-server-output -t 20
iperf 3.1.3
Linux centos-test2.novalocal 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64
Time: Thu, 16 Dec 2021 07:11:10 GMT
Connecting to host 192.168.0.30, port 5201
      Cookie: centos-test2.novalocal.1639638670.73
[  4] local 192.168.0.15 port 38178 connected to 192.168.0.30 port 5201
Starting Test: protocol: UDP, 1 streams, 16 byte blocks, omitting 0 seconds, 20 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.79 MBytes  15.1 Mbits/sec  117610
[  4]   1.00-2.00   sec  1.74 MBytes  14.6 Mbits/sec  113820
[  4]   2.00-3.00   sec  1.69 MBytes  14.2 Mbits/sec  110820
[  4]   3.00-4.00   sec  1.69 MBytes  14.2 Mbits/sec  110990
[  4]   4.00-5.00   sec  1.73 MBytes  14.5 Mbits/sec  113190
[  4]   5.00-6.00   sec  1.76 MBytes  14.8 Mbits/sec  115500
[  4]   6.00-7.00   sec  1.79 MBytes  15.0 Mbits/sec  117480
[  4]   7.00-8.00   sec  1.77 MBytes  14.8 Mbits/sec  115830
[  4]   8.00-9.00   sec  1.77 MBytes  14.8 Mbits/sec  115840
[  4]   9.00-10.00  sec  1.76 MBytes  14.8 Mbits/sec  115440
[  4]  10.00-11.00  sec  1.73 MBytes  14.5 Mbits/sec  113640
[  4]  11.00-12.00  sec  1.72 MBytes  14.5 Mbits/sec  112910
[  4]  12.00-13.00  sec  1.73 MBytes  14.5 Mbits/sec  113200
[  4]  13.00-14.00  sec  1.72 MBytes  14.4 Mbits/sec  112710
[  4]  14.00-15.00  sec  1.72 MBytes  14.4 Mbits/sec  112720
[  4]  15.00-16.00  sec  1.73 MBytes  14.5 Mbits/sec  113650
[  4]  16.00-17.00  sec  1.72 MBytes  14.4 Mbits/sec  112860
[  4]  17.00-18.00  sec  1.73 MBytes  14.5 Mbits/sec  113640
[  4]  18.00-19.00  sec  1.73 MBytes  14.5 Mbits/sec  113560
[  4]  19.00-20.00  sec  1.76 MBytes  14.8 Mbits/sec  115440
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-20.00  sec  34.8 MBytes  14.6 Mbits/sec  0.006 ms  2241/2138201 (0.1%)
[  4] Sent 2138201 datagrams
CPU Utilization: local/sender 28.0% (5.6%u/22.5%s), remote/receiver 3.7% (0.6%u/3.1%s)

Server output:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.15, port 50338
[  5] local 192.168.0.30 port 5201 connected to 192.168.0.15 port 38178
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  1.07 MBytes  9.02 Mbits/sec  0.024 ms  2241/70435 (3.2%)
[  5]   1.00-2.00   sec  1.21 MBytes  10.1 Mbits/sec  0.054 ms  0/79193 (0%)
[  5]   2.00-3.00   sec  1.23 MBytes  10.3 Mbits/sec  0.004 ms  0/80661 (0%)
[  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec  0.009 ms  0/81828 (0%)
[  5]   4.00-5.00   sec  1.22 MBytes  10.2 Mbits/sec  0.006 ms  0/80044 (0%)
[  5]   5.00-6.00   sec  1.15 MBytes  9.68 Mbits/sec  0.006 ms  0/75613 (0%)
[  5]   6.00-7.00   sec  1.11 MBytes  9.34 Mbits/sec  0.022 ms  0/72962 (0%)
[  5]   7.00-8.00   sec  1.12 MBytes  9.43 Mbits/sec  0.005 ms  0/73695 (0%)
[  5]   8.00-9.00   sec  1.13 MBytes  9.44 Mbits/sec  0.021 ms  0/73741 (0%)
[  5]   9.00-10.00  sec  1.12 MBytes  9.41 Mbits/sec  0.021 ms  0/73515 (0%)
[  5]  10.00-11.00  sec  1.11 MBytes  9.34 Mbits/sec  0.006 ms  0/73002 (0%)
[  5]  11.00-12.00  sec  1.12 MBytes  9.36 Mbits/sec  0.014 ms  0/73158 (0%)
[  5]  12.00-13.00  sec  1.11 MBytes  9.35 Mbits/sec  0.004 ms  0/73047 (0%)
[  5]  13.00-14.00  sec  1.13 MBytes  9.47 Mbits/sec  0.046 ms  0/73976 (0%)
[  5]  14.00-15.00  sec  1.12 MBytes  9.38 Mbits/sec  0.004 ms  0/73249 (0%)
[  5]  15.00-16.00  sec  1.14 MBytes  9.56 Mbits/sec  0.011 ms  0/74675 (0%)
[  5]  16.00-17.00  sec  1.16 MBytes  9.72 Mbits/sec  0.027 ms  0/75930 (0%)
[  5]  17.00-18.00  sec  1.13 MBytes  9.52 Mbits/sec  0.007 ms  0/74376 (0%)
[  5]  18.00-19.00  sec  1.11 MBytes  9.31 Mbits/sec  0.005 ms  0/72769 (0%)
[  5]  19.00-20.00  sec  1.13 MBytes  9.45 Mbits/sec  0.006 ms  0/73806 (0%)
[  5]  20.00-21.00  sec  1.56 MBytes  13.1 Mbits/sec  0.012 ms  0/102299 (0%)
[  5]  21.00-22.00  sec  1.55 MBytes  13.0 Mbits/sec  0.026 ms  0/101526 (0%)
[  5]  22.00-23.00  sec  1.56 MBytes  13.1 Mbits/sec  0.018 ms  0/102097 (0%)
[  5]  23.00-24.00  sec  1.98 MBytes  16.6 Mbits/sec  0.036 ms  0/129876 (0%)
[  5]  24.00-25.00  sec  2.26 MBytes  19.0 Mbits/sec  0.004 ms  0/148215 (0%)
[  5]  25.00-25.37  sec   852 KBytes  18.8 Mbits/sec  0.006 ms  0/54513 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-25.37  sec  0.00 Bytes  0.00 bits/sec  0.006 ms  2241/2138201 (0.1%)
[SUM]  0.0-25.4 sec  2241 datagrams received out-of-order

[igsilya]

So my question is does openvswitch kernel module internally caches packet and processes it later?

Some packets can be held inside the kernel, e.g. by the connection tracking module to re-assemble fragmented IP packets. Some packets could be sent to userspace and re-injected back later when the appropriate datapath flow is installed. But I don't think that any of that should be able to buffer 500K packets as in your case and release them later. The packet rate also doesn't seem big enough to cause any performance issues, unless the system has very limited resources or overloaded in some other way. So, something fishy is going on on this setup.

[jshen28]

Thank you very much for reply. I think it is the kernel is buffering packets. I looked at the sysctl configuration and find net.core.netdev_max_backlog is configured to a very large value. and somehow internal port will enqueue received packets to backlog. Try to decrease this value and now packet drop starts to show off.