Pod to Pod traffic is slow on selective path

Environmental Info: K3s Version:

k3s version v1.29.5+k3s1 (4e53a323) go version go1.21.9

Node(s) CPU architecture, OS, and Version:

Linux DNDCICICDTEST0507 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 Servers

k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/723776db4e4f3311c91e0aac1062f5eb0fb9195385ebde44712e44f4055f81e6/bin: - sha256sum: good - links: good System: - /usr/sbin iptables v1.8.7 (nf_tables): ok - swap: should be disabled - routes: default CIDRs 10.42.0.0/16 or 10.43.0.0/16 already routed Limits: - /proc/sys/kernel/keys/root_maxkeys: 1000000 modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-113-generic info: reading kernel config from /boot/config-5.15.0-113-generic ... Generally Necessary: - cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good - /usr/sbin/apparmor_parser apparmor: enabled and tools installed - CONFIG_NAMESPACES: enabled - CONFIG_NET_NS: enabled - CONFIG_PID_NS: enabled - CONFIG_IPC_NS: enabled - CONFIG_UTS_NS: enabled - CONFIG_CGROUPS: enabled - CONFIG_CGROUP_PIDS: enabled - CONFIG_CGROUP_CPUACCT: enabled - CONFIG_CGROUP_DEVICE: enabled - CONFIG_CGROUP_FREEZER: enabled - CONFIG_CGROUP_SCHED: enabled - CONFIG_CPUSETS: enabled - CONFIG_MEMCG: enabled - CONFIG_KEYS: enabled - CONFIG_VETH: enabled (as module) - CONFIG_BRIDGE: enabled (as module) - CONFIG_BRIDGE_NETFILTER: enabled (as module) - CONFIG_IP_NF_FILTER: enabled (as module) - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module) - CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module) - CONFIG_IP_NF_NAT: enabled (as module) - CONFIG_NF_NAT: enabled (as module) - CONFIG_POSIX_MQUEUE: enabled Optional Features: - CONFIG_USER_NS: enabled - CONFIG_SECCOMP: enabled - CONFIG_BLK_CGROUP: enabled - CONFIG_BLK_DEV_THROTTLING: enabled - CONFIG_CGROUP_PERF: enabled - CONFIG_CGROUP_HUGETLB: enabled - CONFIG_NET_CLS_CGROUP: enabled (as module) - CONFIG_CGROUP_NET_PRIO: enabled - CONFIG_CFS_BANDWIDTH: enabled - CONFIG_FAIR_GROUP_SCHED: enabled - CONFIG_RT_GROUP_SCHED: missing - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module) - CONFIG_IP_SET: enabled (as module) - CONFIG_IP_VS: enabled (as module) - CONFIG_IP_VS_NFCT: enabled - CONFIG_IP_VS_PROTO_TCP: enabled - CONFIG_IP_VS_PROTO_UDP: enabled - CONFIG_IP_VS_RR: enabled (as module) - CONFIG_EXT4_FS: enabled - CONFIG_EXT4_FS_POSIX_ACL: enabled - CONFIG_EXT4_FS_SECURITY: enabled - Network Drivers: - "overlay": - CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks): - CONFIG_CRYPTO: enabled - CONFIG_CRYPTO_AEAD: enabled - CONFIG_CRYPTO_GCM: enabled - CONFIG_CRYPTO_SEQIV: enabled - CONFIG_CRYPTO_GHASH: enabled - CONFIG_XFRM: enabled - CONFIG_XFRM_USER: enabled (as module) - CONFIG_XFRM_ALGO: enabled (as module) - CONFIG_INET_ESP: enabled (as module) - CONFIG_INET_XFRM_MODE_TRANSPORT: missing - Storage Drivers: - "overlay": - CONFIG_OVERLAY_FS: enabled (as module) STATUS: pass

Describe the bug:

We have 3 nodes as following

NAME       STATUS   ROLES                       AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
test0507   Ready    control-plane,etcd,master   3h57m   v1.29.5+k3s1   10.130.169.11   <none>        Ubuntu 22.04.4 LTS   5.15.0-113-generic   containerd://1.7.15-k3s1
test0607   Ready    control-plane,etcd,master   3h56m   v1.29.5+k3s1   10.130.169.14   <none>        Ubuntu 22.04.4 LTS   5.15.0-113-generic   containerd://1.7.15-k3s1
test0707   Ready    control-plane,etcd,master   3h56m   v1.29.5+k3s1   10.130.169.15   <none>        Ubuntu 22.04.4 LTS   5.15.0-113-generic   containerd://1.7.15-k3s1

The traffic between pods on test0507 and test0707 is very slow. Setup:

NAME              READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
iperf3-ds-96m58   1/1     Running   0          22s   10.42.1.21   test0707   <none>           <none>
iperf3-ds-sjjvw   1/1     Running   0          22s   10.42.0.19   test0507   <none>           <none>
iperf3-ds-v4stn   1/1     Running   0          22s   10.42.2.20   test0607   <none>           <none>

Traffic from pod on test0507 -> pod on test0707

 k exec -it iperf3-ds-sjjvw -- iperf3 -c 10.42.1.21 -p 5001
Connecting to host 10.42.1.21, port 5001
[  4] local 10.42.0.19 port 49882 connected to 10.42.1.21 port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   498 KBytes  4.08 Mbits/sec   23   1.37 KBytes       
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.37 KBytes       
[  4]   2.00-3.00   sec  62.8 KBytes   514 Kbits/sec    7   1.37 KBytes       
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    5   1.37 KBytes       
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    5   1.37 KBytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    4   2.73 KBytes       
[  4]   6.00-7.00   sec   188 KBytes  1.54 Mbits/sec   13   9.56 KBytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    2   1.37 KBytes       
[  4]   8.00-9.00   sec  62.8 KBytes   514 Kbits/sec   18   1.37 KBytes       
[  4]   9.00-10.00  sec  62.8 KBytes   514 Kbits/sec   12   5.46 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   875 KBytes   717 Kbits/sec   90             sender
[  4]   0.00-10.00  sec   624 KBytes   511 Kbits/sec                  receiver

Traffic from pod on test0607 -> pod on test0707

 k exec -it iperf3-ds-v4stn -- iperf3 -c 10.42.1.21 -p 5001
Connecting to host 10.42.1.21, port 5001
[  4] local 10.42.2.20 port 60654 connected to 10.42.1.21 port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   169 MBytes  1.41 Gbits/sec  110    116 KBytes       
[  4]   1.00-2.00   sec   136 MBytes  1.14 Gbits/sec    4    414 KBytes       
[  4]   2.00-3.00   sec   187 MBytes  1.57 Gbits/sec    1    668 KBytes       
[  4]   3.00-4.00   sec   182 MBytes  1.53 Gbits/sec  178    698 KBytes       
[  4]   4.00-5.00   sec   151 MBytes  1.27 Gbits/sec  364    317 KBytes       
[  4]   5.00-6.00   sec   145 MBytes  1.22 Gbits/sec    6    358 KBytes       
[  4]   6.00-7.00   sec   166 MBytes  1.39 Gbits/sec  165    427 KBytes       
[  4]   7.00-8.00   sec  93.5 MBytes   784 Mbits/sec   36    117 KBytes       
[  4]   8.00-9.00   sec   147 MBytes  1.24 Gbits/sec    2    475 KBytes       
[  4]   9.00-10.00  sec   171 MBytes  1.44 Gbits/sec  105    347 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.51 GBytes  1.30 Gbits/sec  971             sender
[  4]   0.00-10.00  sec  1.51 GBytes  1.30 Gbits/sec                  receiver

Traffic from node to node is fine though test0507 to test0707

------------------------------------------------------------
Client connecting to 10.130.169.15, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 10.130.169.11 port 32828 connected with 10.130.169.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0041 sec  6.14 GBytes  5.27 Gbits/sec

Steps To Reproduce:

Install k3s on 3 nodes

Deploy a ds with the below yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: iperf3-ds
namespace: network
spec:
selector:
matchLabels:
  app: iperf3
template:
metadata:
  labels:
    app: iperf3
spec:
  containers:
  - name: iperf3
    image: leodotcloud/swiss-army-knife
    ports:
    - containerPort: 5201

Check network bandwidth

k3s-io / k3s

Pod to Pod traffic is slow on selective path #10574