search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.62k stars 2.31k forks source link

k3s agent flannel exit failed #2547

Closed hechengjie closed 3 years ago

hechengjie commented 3 years ago

INFO[2020-11-16T16:06:28.137335000Z] Starting k3s agent v1.19.3+k3s3 (0e4fbfef)
INFO[2020-11-16T16:06:28.139898340Z] Module overlay was already loaded
INFO[2020-11-16T16:06:28.140011040Z] Module nf_conntrack was already loaded
INFO[2020-11-16T16:06:28.140067660Z] Module br_netfilter was already loaded
INFO[2020-11-16T16:06:28.140116600Z] Module iptable_nat was already loaded
INFO[2020-11-16T16:06:28.141066240Z] Running load balancer 127.0.0.1:32789 -> [10.68.7.181:6443] INFO[2020-11-16T16:06:29.152893520Z] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log INFO[2020-11-16T16:06:29.153728540Z] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd INFO[2020-11-16T16:06:30.161418680Z] Containerd is now running
INFO[2020-11-16T16:07:07.604404020Z] Connecting to proxy url="wss://10.68.7.181:6443/v1-k3s/connect" WARN[2020-11-16T16:07:07.689145340Z] Disabling CPU quotas due to missing cpu.cfs_period_us INFO[2020-11-16T16:07:07.689293660Z] Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/63dbbe8462074f773c2a00caa9a9a87684fcff868c7e6bf54b4433af70608590/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=/run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ambarella --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --kubelet-cgroups=/systemd/user.slice/user-0.slice --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/k3s-resolv.conf --runtime-cgroups=/systemd/user.slice/user-0.slice --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key INFO[2020-11-16T16:07:07.697543240Z] Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=ambarella --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables W1116 16:07:07.704734 2220 server.go:226] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP. Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet. Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed. INFO[2020-11-16T16:07:07.792701540Z] Node CIDR assigned for: ambarella
I1116 16:07:07.797122 2220 flannel.go:92] Determining IP address of default interface I1116 16:07:07.814142 2220 flannel.go:105] Using interface with name eth0 and address 172.16.195.153 INFO[2020-11-16T16:07:07.817901760Z] labels have already set on node: ambarella
I1116 16:07:07.849499 2220 kube.go:117] Waiting 10m0s for node controller to sync I1116 16:07:07.849652 2220 kube.go:300] Starting kube subnet manager I1116 16:07:07.856689 2220 node.go:136] Successfully retrieved node IP: 172.16.195.153 I1116 16:07:07.868413 2220 server_others.go:112] kube-proxy node IP is an IPv4 address (172.16.195.153), assume IPv4 operation I1116 16:07:07.936675 2220 server.go:407] Version: v1.19.3+k3s3 I1116 16:07:07.960025 2220 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt I1116 16:07:07.988073 2220 server_others.go:187] Using iptables Proxier. I1116 16:07:07.992003 2220 server.go:650] Version: v1.19.3+k3s3 I1116 16:07:07.998463 2220 conntrack.go:52] Setting nf_conntrack_max to 131072 I1116 16:07:08.059460 2220 config.go:315] Starting service config controller I1116 16:07:08.059521 2220 shared_informer.go:240] Waiting for caches to sync for service config I1116 16:07:08.059623 2220 config.go:224] Starting endpoint slice config controller I1116 16:07:08.059648 2220 shared_informer.go:240] Waiting for caches to sync for endpoint slice config I1116 16:07:08.160271 2220 shared_informer.go:247] Caches are synced for endpoint slice config I1116 16:07:08.160271 2220 shared_informer.go:247] Caches are synced for service config I1116 16:07:08.205972 2220 network_policy_controller.go:149] Starting network policy controller W1116 16:07:08.518939 2220 iptables.go:556] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 4: iptables v1.8.3 (nf_tables): CHAIN_ADD failed (No such file or directory): chain OUTPUT E1116 16:07:08.674874 2220 proxier.go:866] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking rule: exit status 2: iptables v1.8.3 (nf_tables): Couldn't find match `conntrack'

Try `iptables -h' or 'iptables --help' for more information. I1116 16:07:08.674930 2220 proxier.go:850] Sync failed; retrying in 30s I1116 16:07:08.858552 2220 kube.go:124] Node controller sync successful I1116 16:07:08.858679 2220 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false FATA[2020-11-16T16:07:08.878088680Z] flannel exited: operation not supported

I run k3s agent at arm linux. Because of the flash is too small, i mount the rootfs by nfs. Then the eth0 cannot be operated. Is it fata caused by eth device or the iptables ?

brandond commented 3 years ago

You appear to be missing some iptables kernels modules:

W1116 16:07:08.518939 2220 iptables.go:556] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 4: iptables v1.8.3 (nf_tables): CHAIN_ADD failed (No such file or directory): chain OUTPUT
E1116 16:07:08.674874 2220 proxier.go:866] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking rule: exit status 2: iptables v1.8.3 (nf_tables): Couldn't find match `conntrack'

I'm not sure what flannel is attempting to do that is not supported by your interface, that might require additional tracing to determine:

FATA[2020-11-16T16:07:08.878088680Z] flannel exited: operation not supported
hechengjie commented 3 years ago

i miss vxlan.ko. Close it.