flannel vxlan loss of 30% of performance

In my environment, I use the flannel vxlan mode for cross-host pod access performance testing, and there is about a 30% bandwidth drop. The following is my test data:

Cross-Bare Metal Testing： node1:

iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local node1 port 5001 connected with node2 port 46028
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  32.1 GBytes  9.19 Gbits/sec

node2：

iperf -c node1 -t30 -i5 -e
------------------------------------------------------------
Client connecting to node1, TCP port 5001 with pid 77458
Write buffer size:  128 KByte
TCP window size: 2.64 MByte (default)
------------------------------------------------------------
[  3] local node2 port 46028 connected with node1 port 5001 (ct=0.42 ms)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  3] 0.00-5.00 sec  5.33 GBytes  9.16 Gbits/sec  43661/0          0      449K/473 us  2419760.93
[  3] 5.00-10.00 sec  5.31 GBytes  9.12 Gbits/sec  43507/0        635      692K/296 us  3853073.99
[  3] 10.00-15.00 sec  5.34 GBytes  9.17 Gbits/sec  43717/0        765      680K/440 us  2604579.37
[  3] 15.00-20.00 sec  5.42 GBytes  9.32 Gbits/sec  44429/0        289      933K/377 us  3089335.75
[  3] 20.00-25.00 sec  5.38 GBytes  9.25 Gbits/sec  44086/0        278     1088K/339 us  3409109.26
[  3] 25.00-30.00 sec  5.31 GBytes  9.12 Gbits/sec  43509/0        350      943K/577 us  1976711.14
[  3] 0.00-30.00 sec  32.1 GBytes  9.19 Gbits/sec  262909/0       2317       -1K/417 us  2754586.44

Cross-bare-metal pod testing： pod1 on node1：

iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.244.3.134 port 5001 connected with 10.244.4.31 port 53918
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  22.9 GBytes  6.56 Gbits/sec

pod2 on node2:

iperf -c 10.244.3.134 -t 30 -i5 -e
------------------------------------------------------------
Client connecting to 10.244.3.134, TCP port 5001 with pid 49
Write buffer size:  128 KByte
TCP window size:  526 KByte (default)
------------------------------------------------------------
[  3] local 10.244.4.31 port 53918 connected with 10.244.3.134 port 5001 (ct=0.72 ms)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  3] 0.00-5.00 sec  3.63 GBytes  6.24 Gbits/sec  29746/0          0      161K/438 us  1780304.89
[  3] 5.00-10.00 sec  3.68 GBytes  6.32 Gbits/sec  30127/0       2615      311K/441 us  1790841.79
[  3] 10.00-15.00 sec  3.96 GBytes  6.80 Gbits/sec  32437/0       3398      533K/608 us  1398546.86
[  3] 15.00-20.00 sec  4.32 GBytes  7.43 Gbits/sec  35413/0       3343      606K/614 us  1511939.00
[  3] 20.00-25.00 sec  3.73 GBytes  6.41 Gbits/sec  30565/0       3160      716K/642 us  1248042.27
[  3] 25.00-30.00 sec  3.59 GBytes  6.17 Gbits/sec  29441/0       5504      653K/510 us  1513290.49
[  3] 0.00-30.00 sec  22.9 GBytes  6.56 Gbits/sec  187729/0      18020       -1K/542 us  1513244.28

My Environment

Flannel version: v0.22.2
Backend used (e.g. vxlan or udp): vxlan
Kubernetes version (if used): v1.24.10, six node

kube-flannel-cfg:

cni-conf.json: |
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
net-conf.json: |
{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan"
  }
}

Flannel ds start command: /opt/bin/flanneld --ip-masq --kube-subnet-mgr
Operating System and version: CentOS7.9 3.10.0-1160.el7.x86_64
VXLAN offlad is on
Network Card: bonded mode=1 miimon=100
Iperf: 2.0.13
My Doubt

1.Flannel vxlan mode, is the above performance loss normal? 2.How can I adjust flannel vxlan parameters to reduce performance loss?

flannel-io / flannel

flannel vxlan loss of 30% of performance #1816

My Environment

My Doubt