rajch
commented
1 week ago From what I can make out, weave is crashing on five nodes while trying to add a dummy link for setting the bridge MTU. Before we try to diagnose the root cause, please help me confirm the problem.
- Please run
kubectl get pods -n kube-system -o wideand paste the results here. This will show the nodes where the weave pods are failing. - Next, choose any two failing weave pods, and run
kubectl logs -n kube-system <pod name> weaveon both of them, and paste the results here. - On the two nodes whose weave logs you pasted in the last step, please run
lsmod, and paste the results here, indicating which node is which.
Thanks in advance, and sorry for the trouble.
zz0350
Expected Outcome: pod/weave-net-lxzfx 2/2 Running 7 (26h ago) 3d3h
Actual Outcome:
cyx@node2:/$ kubectl get all -n kube-system NAME READY STATUS RESTARTS AGE pod/coredns-7db6d8ff4d-b2v92 1/1 Running 34 (26h ago) 32d pod/coredns-7db6d8ff4d-fkffx 1/1 Running 34 (26h ago) 32d pod/etcd-node2 1/1 Running 62 (26h ago) 32d pod/kube-apiserver-node2 1/1 Running 58 (26h ago) 32d pod/kube-controller-manager-node2 1/1 Running 38 (26h ago) 32d pod/kube-proxy-22bmm 1/1 Running 0 54m pod/kube-proxy-gzwbw 1/1 Running 3 (26h ago) 3d3h pod/kube-proxy-kbw84 1/1 Running 1 (35m ago) 61m pod/kube-proxy-phjgc 1/1 Running 0 55m pod/kube-proxy-pq6rx 1/1 Running 0 54m pod/kube-proxy-rtplg 1/1 Running 0 54m pod/kube-proxy-vbdtp 1/1 Running 5 (2d3h ago) 4d23h pod/kube-proxy-wghts 1/1 Running 35 (26h ago) 32d pod/kube-scheduler-node2 1/1 Running 39 (26h ago) 32d pod/weave-net-5rfjt 2/2 Running 72 (26h ago) 32d pod/weave-net-7cz9p 1/2 CrashLoopBackOff 15 (2m31s ago) 54m pod/weave-net-cswgw 1/2 CrashLoopBackOff 15 (118s ago) 54m pod/weave-net-fr6s9 1/2 CrashLoopBackOff 20 (4m20s ago) 61m pod/weave-net-lxzfx 2/2 Running 7 (26h ago) 3d3h pod/weave-net-mcrzw 2/2 Running 16 (2d3h ago) 4d23h pod/weave-net-q59j8 1/2 CrashLoopBackOff 15 (2m34s ago) 54m pod/weave-net-wgh9w 1/2 CrashLoopBackOff 15 (2m12s ago) 55m Reproduction: In my cluster, there are already 2 worker nodes (with the weave plugin working correctly) and one control plane (with weave working correctly). However, the weave plugin on the subsequent 5 worker nodes all went into CrashLoopBackOff.
Hardware: The 5 worker nodes added later are all Raspberry Pi 4 Model B, while the first 3 successful ones included one Raspberry Pi 4 Model B.
Weave Configuration: Applied directly using kubectl apply -f net.yaml. Kubernetes version is 1.30.
Weave version:
INFO: 2024/11/09 13:32:15.456124 weave 2.8.9 Docker: For the first 3 successful ones: Docker 27.0.3, installed from a package. For the next 5 failed ones: Docker 27.3.1, installed using the apt repository.
System Information:
uname -a: (false) Linux node5 5.15.0-1061-raspi weaveworks#64-Ubuntu SMP PREEMPT Wed Aug 7 14:41:30 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux (succeed) Linux node2 6.8.0-1013-raspi #14-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 2 15:14:53 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux (succeed) Linux node3 6.8.0-1013-raspi #14-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 2 15:14:53 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux (false) Linux node11 5.15.0-1061-raspi weaveworks#64-Ubuntu SMP PREEMPT Wed Aug 7 14:41:30 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux Kubectl version:
cyx@node2:/$ kubectl version Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0
cyx@node3:/proc/sys/net/core$ kubectl version Client Version: v1.30.6 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 error: unable to parse the server version: invalid character '<' looking for beginning of value
cyx@node5:/etc$ kubectl version Client Version: v1.30.6 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 error: unable to parse the server version: invalid character '<' looking for beginning of value
cyx@node11:~/docker_images$ kubectl version Client Version: v1.30.6 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 error: unable to parse the server version: invalid character '<' looking for beginning of value Weave Logs:
cyx@node2:/$ kubectl logs weave-net-7cz9p -n kube-system Defaulted container "weave" out of: weave, weave-npc, weave-init (init) iptables backend mode: nft DEBU: 2024/11/09 13:47:28.706109 [kube-peers] Checking peer "ee:78:5a:ee:17:61" against list &{[{4e:ea:7b:ce:a4:2d node2} {6a:b7:c0:3c:ac:fb node4} {6e:1c:d3:1c:2a:0a node3}]} Peer not in list; removing persisted data INFO: 2024/11/09 13:47:29.528736 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=8 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:ee:78:5a:ee:17:61 nickname:node10 no-dns:true no-masq-local:true port:6783] INFO: 2024/11/09 13:47:29.528893 weave 2.8.9 FATA: 2024/11/09 13:47:29.847512 creating dummy interface: operation not supported
cyx@node2:/$ kubectl logs weave-net-7cz9p -n kube-system weave iptables backend mode: nft DEBU: 2024/11/09 13:47:28.706109 [kube-peers] Checking peer "ee:78:5a:ee:17:61" against list &{[{4e:ea:7b:ce:a4:2d node2} {6a:b7:c0:3c:ac:fb node4} {6e:1c:d3:1c:2a:0a node3}]} Peer not in list; removing persisted data INFO: 2024/11/09 13:47:29.528736 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=8 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:ee:78:5a:ee:17:61 nickname:node10 no-dns:true no-masq-local:true port:6783] INFO: 202