search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.91k stars 2.33k forks source link

The newly added k3s agent nodes are unable to access the network of certain namespaces, whereas the initially added nodes are functioning without issues. #9098

Closed gaozuo closed 9 months ago

gaozuo commented 9 months ago

Environmental Info: K3s Version: 

[root@MH-HR-sscphjsk ~]# k3s -v
k3s version v1.27.4+k3s1 (36645e73)
go version go1.20.6

Node(s) CPU architecture, OS, and Version: 

Linux MH-HR-app6 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-xxcj-app1 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-app5 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-xxcj-app2-2 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-xxcj-app2-3 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-interface1 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux zuzhi-GIS-app-3 4.19.90-23.25.v2101.ky10.x86_64 #1 SMP Tue Jun 21 19:41:22 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
Linux MH-HR-sscphjsk 3.10.0-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

[root@MH-HR-sscphjsk ~]# kubectl get node -o wide
NAME                STATUS   ROLES                  AGE    VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                   KERNEL-VERSION                    CONTAINER-RUNTIME
mh-hr-app6          Ready    <none>                 5d4h   v1.27.4+k3s1   10.201.225.80    <none>        CentOS Linux 7 (Core)                      3.10.0-1160.76.1.el7.x86_64       containerd://1.7.1-k3s1
mh-hr-xxcj-app1     Ready    <none>                 62d    v1.27.4+k3s1   10.201.225.124   <none>        CentOS Linux 7 (Core)                      3.10.0-1160.83.1.el7.x86_64       containerd://1.7.1-k3s1
mh-hr-app5          Ready    <none>                 16d    v1.27.4+k3s1   10.201.225.79    <none>        CentOS Linux 7 (Core)                      3.10.0-1160.76.1.el7.x86_64       containerd://1.7.1-k3s1
mh-hr-xxcj-app2-2   Ready    <none>                 67d    v1.27.4+k3s1   10.201.225.126   <none>        CentOS Linux 7 (Core)                      3.10.0-1160.83.1.el7.x86_64       containerd://1.7.1-k3s1
mh-hr-xxcj-app2-3   Ready    <none>                 119d   v1.27.4+k3s1   10.201.225.127   <none>        CentOS Linux 7 (Core)                      3.10.0-1160.83.1.el7.x86_64       containerd://1.7.1-k3s1
mh-hr-interface1    Ready    <none>                 67d    v1.27.4+k3s1   10.201.225.121   <none>        CentOS Linux 7 (Core)                      3.10.0-1160.83.1.el7.x86_64       containerd://1.7.1-k3s1
zuzhi-gis-app-3     Ready    <none>                 119d   v1.27.4+k3s1   10.201.225.219   <none>        Kylin Linux Advanced Server V10 (Tercel)   4.19.90-23.25.v2101.ky10.x86_64   containerd://1.7.1-k3s1
mh-hr-sscphjsk      Ready    control-plane,master   119d   v1.27.4+k3s1   10.201.225.130   <none>        CentOS Linux 7 (Core)                      3.10.0-1160.92.1.el7.x86_64       containerd://1.7.1-k3s1

Describe the bug:

When we initially set up the cluster, we used 1 master node and 5 agent nodes, which operated smoothly and stably for a while. Later, we added two more agent nodes (mh-hr-app5, mh-hr-app6). Now, when new pods are scheduled on these two nodes, they are unable to access Kafka and Zookeeper services in the Confluent namespace. However, access to HTTP-type services remains normal across all namespaces. This leads us to suspect that the network configuration of these newly added agent nodes might be inconsistent with the network configuration script used during the initial cluster installation.

Steps To Reproduce:

export INSTALL_K3S_MIRROR=cn
export K3S_URL=https://10.201.225.130:6443
export INSTALL_K3S_VERSION=v1.27.4+k3s1
export K3S_TOKEN=K10c46dd38636c4073c7d202599f35849e08b53e719f853d89d9134245365e6e7f7::server:1ef415ad804694b2809312d340cc6e34
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | sh -

Additional context / logs:

[root@MH-HR-sscphjsk ~]# kubectl get pods -n confluent -o wide
NAME                                  READY   STATUS    RESTARTS        AGE   IP            NODE               NOMINATED NODE   READINESS GATES
confluent-operator-7867b75d94-gkr6l   1/1     Running   0               18m   10.42.5.183   mh-hr-app5         <none>           <none>
zookeeper-0                           1/1     Running   0               18m   10.42.4.221   mh-hr-interface1   <none>           <none>
kafka-0                               1/1     Running   0               18m   10.42.4.222   mh-hr-interface1   <none>           <none>
kafkarestproxy-0                      1/1     Running   0               18m   10.42.5.185   mh-hr-app5         <none>           <none>
ksqldb-0                              1/1     Running   1 (18m ago)     18m   10.42.6.175   mh-hr-xxcj-app1    <none>           <none>
controlcenter-0                       1/1     Running   0               18m   10.42.0.32    mh-hr-sscphjsk     <none>           <none>
connect-0                             0/1     Running   6 (3m45s ago)   18m   10.42.5.184   mh-hr-app5         <none>           <none>
schemaregistry-0                      0/1     Running   7 (5m56s ago)   18m   10.42.5.186   mh-hr-app5         <none>           <none>
connect-worker0-0                     0/1     Running   6 (3m21s ago)   18m   10.42.5.187   mh-hr-app5         <none>           <none>
[root@MH-HR-sscphjsk ~]# kubectl get pods -n flink-cluster -o wide
NAME                                       READY   STATUS    RESTARTS        AGE     IP            NODE                NOMINATED NODE   READINESS GATES
sql-person-c-7cfbfd7494-sf2rc              1/1     Running   0               18d     10.42.4.155   mh-hr-interface1    <none>           <none>
sql-person-b-86b7f966d9-h5dch              1/1     Running   2 (9d ago)      20d     10.42.2.98    zuzhi-gis-app-3     <none>           <none>
sql-baseemp-745b8b45c4-wl9cr               1/1     Running   4 (8d ago)      19d     10.42.6.115   mh-hr-xxcj-app1     <none>           <none>
sql-person-a-5455d649dc-6cx48              1/1     Running   3 (2d13h ago)   20d     10.42.4.142   mh-hr-interface1    <none>           <none>
sql-common-b-5657969474-z88hl              1/1     Running   5 (2d7h ago)    20d     10.42.1.165   mh-hr-xxcj-app2-3   <none>           <none>
sql-common-b-taskmanager-5-1               1/1     Running   0               2d7h    10.42.1.216   mh-hr-xxcj-app2-3   <none>           <none>
sql-common-a-taskmanager-6-2               1/1     Running   0               2d      10.42.1.221   mh-hr-xxcj-app2-3   <none>           <none>
sql-common-a-taskmanager-6-1               1/1     Running   0               2d      10.42.1.222   mh-hr-xxcj-app2-3   <none>           <none>
sql-common-a-6977f86c77-tsbct              1/1     Running   95 (2d ago)     17d     10.42.1.181   mh-hr-xxcj-app2-3   <none>           <none>
flink-kubernetes-operator-f4bbff6-fxkr4    2/2     Running   0               23d     10.42.0.254   mh-hr-sscphjsk      <none>           <none>
sql-person-5dcc7556c8-g4bx4                1/1     Running   0               10h     10.42.4.210   mh-hr-interface1    <none>           <none>
sql-person-taskmanager-1-1                 1/1     Running   0               10h     10.42.4.211   mh-hr-interface1    <none>           <none>
sql-person-taskmanager-1-2                 1/1     Running   0               10h     10.42.4.212   mh-hr-interface1    <none>           <none>
sql-project-performance-taskmanager-2-1    1/1     Running   0               6h47m   10.42.7.57    mh-hr-app6          <none>           <none>
sql-organ-taskmanager-37-2                 1/1     Running   0               4h4m    10.42.2.128   zuzhi-gis-app-3     <none>           <none>
sql-organ-taskmanager-37-3                 1/1     Running   0               4h4m    10.42.2.129   zuzhi-gis-app-3     <none>           <none>
sql-organ-75c65d79f5-g8n5k                 1/1     Running   128 (72m ago)   2d1h    10.42.2.83    zuzhi-gis-app-3     <none>           <none>
sql-project-performance-taskmanager-7-1    1/1     Running   0               8m35s   10.42.5.188   mh-hr-app5          <none>           <none>
sql-project-performance-777d59f95f-pddn6   1/1     Running   8 (8m26s ago)   7h46m   10.42.7.54    mh-hr-app6          <none>           <none>
[root@MH-HR-sscphjsk ~]# kubectl exec -it sql-project-performance-taskmanager-7-1 -n flink-cluster -- bash
root@sql-project-performance-taskmanager-7-1:/opt/flink# curl -v http://kafka.confluent:9092
*   Trying 10.42.4.222:9092...
* connect to 10.42.4.222 port 9092 failed: Connection timed out
* Failed to connect to kafka.confluent port 9092 after 127326 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to kafka.confluent port 9092 after 127326 ms: Connection timed out
root@sql-project-performance-taskmanager-7-1:/opt/flink# curl -v http://zookeeper.confluent:2181
*   Trying 10.42.4.221:2181...
^C
-------------------------
root@sql-project-performance-taskmanager-7-1:/opt/flink# curl -v http://controlcenter.confluent:9021
*   Trying 10.42.0.32:9021...
* Connected to controlcenter.confluent (10.42.0.32) port 9021 (#0)
> GET / HTTP/1.1
> Host: controlcenter.confluent:9021
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 27 Dec 2023 14:49:34 GMT
< X-Content-Type-Options: nosniff
< X-Confluent-Control-Center-Version: 7.4.0
< X-Confluent-Control-Session: f6528f7b-281a-43eb-8030-ba15409fac62
< Strict-Transport-Security: max-age=31536000
< Cache-Control: no-store
< X-Frame-Options: SAMEORIGIN
< Last-Modified: Mon, 01 May 2023 04:47:08 GMT
< Content-Type: text/html
< Accept-Ranges: bytes
< Vary: Accept-Encoding, User-Agent
< Content-Length: 7119
<
--------------------------
[root@MH-HR-sscphjsk ~]# kubectl exec -it sql-person-taskmanager-1-2 -n flink-cluster -- bash
root@sql-person-taskmanager-1-2:/opt/flink# curl -v http://kafka.confluent:9092
*   Trying 10.42.4.222:9092...
* Connected to kafka.confluent (10.42.4.222) port 9092 (#0)
> GET / HTTP/1.1
> Host: kafka.confluent:9092
> User-Agent: curl/7.81.0
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

root@sql-person-taskmanager-1-2:/opt/flink# curl -v http://controlcenter.confluent:9021
*   Trying 10.42.0.32:9021...
* Connected to controlcenter.confluent (10.42.0.32) port 9021 (#0)
> GET / HTTP/1.1
> Host: controlcenter.confluent:9021
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 27 Dec 2023 14:52:27 GMT
< X-Content-Type-Options: nosniff
< X-Confluent-Control-Center-Version: 7.4.0
< X-Confluent-Control-Session: f6528f7b-281a-43eb-8030-ba15409fac62
< Strict-Transport-Security: max-age=31536000
< Cache-Control: no-store
< X-Frame-Options: SAMEORIGIN
< Last-Modified: Mon, 01 May 2023 04:47:08 GMT
< Content-Type: text/html
< Accept-Ranges: bytes
< Vary: Accept-Encoding, User-Agent
< Content-Length: 7119
[Original agent node iptable--list configuration.log](https://github.com/k3s-io/k3s/files/13782385/Original.agent.node.iptable--list.configuration.log)
[New agent node iptable--list configuration.log](https://github.com/k3s-io/k3s/files/13782386/New.agent.node.iptable--list.configuration.log)
brandond commented 9 months ago

Cluster networking is not namespaced. I suspect that your problem is related to inter-node CNI traffic - as you noted, the problem has more to do with which nodes the pods are running on. Confirm that the correct ports are open on the security groups for the nodes you just added, for whatever flannel backend you're using.

gaozuo commented 9 months ago

The issue has been resolved. It was evident from all scenarios that the traffic between nodes was abnormal. We later found out that the network team had implemented a blocking strategy without our knowledge. After readjusting the network rules, the cluster is now functioning well.