search

cloudnativer / kube-install

一键安装k8s(kubernetes)：二進位離線方式部署，支持定時安裝、添加與銷毀node、銷毀與修復master、一鍵卸載集羣等。Install k8s (kubernetes): one click offline installation of highly available multiple kubernetes cluster, supports schedule installation, addition of nodes, rebuild of kubernetes master, and uninstallation of clusters.
Apache License 2.0
422 stars 118 forks source link

Kubernetes node add failed! #36

Open CYS158 opened 2 years ago

CYS158 commented 2 years ago

add node failed logs：

[Info] 2022-06-16 16:58:14.38258995 +0800 CST m=+1655.018892666 Adding kubernetes node, please wait ... 

Kubernetes Cluster Label: K8s-Cluster1
Kubernetes Node: 8.8.88.86
SSH Operation Port: 22
Operating System Type: rhel7
Automatically Upgrade OS Kernel: NO
System User for Operation: root

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : 0.Distributing deployment files to target host, please wait...] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : copy] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : copy] *** changed: [8.8.88.86]

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 1.Create base package directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 2.Perform installation pretreatment] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : shell] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 3.Turn off firewall] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : service] *** fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service ufw: host"} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : service] *** fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service SuSEfirewall2: host"} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 4.Turn off selinux] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 5.Turn off postfix] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 6.Set system limit] ** changed: [8.8.88.86] => (item= soft nofile 65536) changed: [8.8.88.86] => (item= hard nofile 65536) changed: [8.8.88.86] => (item= soft nproc 65536) changed: [8.8.88.86] => (item= hard nproc 65536) changed: [8.8.88.86] => (item= soft memlock unlimited) changed: [8.8.88.86] => (item=* hard memlock unlimited)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 7.Set ipvs_modules] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 8.Run ipvs_modules] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 9.Add sysctl parameter configuration] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 10.Set sysctl parameter configuration] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 11.Create journal directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 12.Create journal.conf.d directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 13.Add prophet file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 14.Start systemd-journald service] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 15.Install TTY software] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : copy] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 16.Create resolv config file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 17.Create k8s cert directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 18.Decompress k8s software package] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 19.Decompress cfssl software package] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 20.Decompress helm software package] *** changed: [8.8.88.86]

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 1.Cleaning up garbage files left in history.] *** fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : service] *** fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service kube-proxy: host"} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : service] *** fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service containerd: host"} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : shell] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86] => (item=containerd) ok: [8.8.88.86] => (item=containerd-shim) ok: [8.8.88.86] => (item=containerd-shim-runc-v1) ok: [8.8.88.86] => (item=containerd-shim-runc-v2) ok: [8.8.88.86] => (item=nerdctl) ok: [8.8.88.86] => (item=ctr) ok: [8.8.88.86] => (item=runc)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 2.Create CNI plug-in directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 3.Decompress CNI plug-in package] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : unarchive] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 4.Create runtime config dir] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 5.Decompress runtime software package] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : template] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 6.Create containerd service] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 7.Create ipvsinit service] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 8.Create ipvsinit shell] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 9.Enable ipvsinit service] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 10.Start runtime service] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 11.Decompress runtime images package] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 12.Load runtime images] *** changed: [8.8.88.86] => (item=metrics-server-amd64.tar) changed: [8.8.88.86] => (item=metrics-scraper.tar) changed: [8.8.88.86] => (item=heapster-amd64.tar) changed: [8.8.88.86] => (item=pause-amd64.tar) changed: [8.8.88.86] => (item=traefik.tar) changed: [8.8.88.86] => (item=alpine.tar) changed: [8.8.88.86] => (item=kubernetes-dashboard-v2.4.0-amd64.tar) changed: [8.8.88.86] => (item=coredns.tar) changed: [8.8.88.86] => (item=registry.tar) changed: [8.8.88.86] => (item=calico-cni.tar) changed: [8.8.88.86] => (item=calico-kube-controllers.tar) changed: [8.8.88.86] => (item=calico-node.tar) changed: [8.8.88.86] => (item=calico-pod2daemon-flexvol.tar) changed: [8.8.88.86] => (item=flannel-cni-plugin.tar) changed: [8.8.88.86] => (item=flannel.tar) changed: [8.8.88.86] => (item=kube-router.tar) changed: [8.8.88.86] => (item=weave-kube.tar) changed: [8.8.88.86] => (item=weave-npc.tar) changed: [8.8.88.86] => (item=cilium.tar) changed: [8.8.88.86] => (item=cilium-operator-generic.tar)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 13.Set runtime images tag and create pushsysimages shell] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : shell] *** changed: [8.8.88.86] => (item=metrics-server-amd64:v0.5.0) changed: [8.8.88.86] => (item=metrics-scraper:v1.0.7) changed: [8.8.88.86] => (item=heapster-amd64:v1.5.4) changed: [8.8.88.86] => (item=pause-amd64:3.5) changed: [8.8.88.86] => (item=traefik:v2.0.7) changed: [8.8.88.86] => (item=alpine:3.6) changed: [8.8.88.86] => (item=kubernetes-dashboard-amd64:v2.4.0) changed: [8.8.88.86] => (item=coredns:1.3.1) changed: [8.8.88.86] => (item=registry:v2.7.1) changed: [8.8.88.86] => (item=calico-node:v3.19.3) changed: [8.8.88.86] => (item=calico-pod2daemon-flexvol:v3.19.3) changed: [8.8.88.86] => (item=calico-cni:v3.19.3) changed: [8.8.88.86] => (item=calico-kube-controllers:v3.19.3) changed: [8.8.88.86] => (item=flannel-cni-plugin:v1.0.0) changed: [8.8.88.86] => (item=flannel:v0.15.1) changed: [8.8.88.86] => (item=kube-router:v1.3.2) changed: [8.8.88.86] => (item=weave-kube:2.8.1) changed: [8.8.88.86] => (item=weave-npc:2.8.1) changed: [8.8.88.86] => (item=cilium:v1.9.0) changed: [8.8.88.86] => (item=cilium-operator-generic:v1.9.0)

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 1.Distribution kubelet cert] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 2.Clean up invalid certificates] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 3.Copy kubelet config and shell file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : copy] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 4.Set IPVS rules] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 5.Create /home/kube-install/k8s/kubernetes directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 6.Create kubelet service startup file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 7.Start kubelet service] *** changed: [8.8.88.86]

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 1.Create kube-proxy.config.yaml configuration] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 2.Create /home/kube-install/k8s/kubernetes/kube-proxy directory] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 3.Create kube-proxy service startup file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 4.Start kube-proxy service] *** changed: [8.8.88.86]

PLAY [master1] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 1.Enable csr-crb configuration] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 2.Wait 90s] *** ok: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 3.Approve node csr] *** fatal: [8.8.88.81]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/kubectl get csr | awk '/Pending/ {print $1}' | xargs kubectl certificate approve", "delta": "0:00:00.118209", "end": "2022-06-16 17:06:44.673905", "msg": "non-zero return code", "rc": 123, "start": "2022-06-16 17:06:44.555696", "stderr": "error: one or more CSRs must be specified as or -f ", "stderr_lines": ["error: one or more CSRs must be specified as or -f "], "stdout": "", "stdout_lines": []} ...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 4.Wait 30s] *** ok: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 5. Setting scheduling label for k8s node] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] *** changed: [8.8.88.81]

PLAY [addnode] *****

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : Create reboot config file] *** changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : Congratulations, node successfully joined kubernetes cluster! "The operating system will automatically restart to take effect on the cluster configuration."] *** ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : shell] *** changed: [8.8.88.86]

PLAY RECAP ***** 8.8.88.81 : ok=11 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
8.8.88.86 : ok=77 changed=62 unreachable=0 failed=0 skipped=0 rescued=0 ignored=5

[Error] 2022-06-16 17:13:19.840843904 +0800 CST m=+2560.477146626 Kubernetes node add failed!

cloudnativer commented 2 years ago

What is the detailed version of the operating system, the CNI plug-in selected, and the kube-install version? Is there any restriction on the network from the k8s-master to the k8s-node?

cloudnativer commented 2 years ago

Has this k8s-node performed the action of adding repeatedly? You can try delnode and addnode again.

CYS158 commented 2 years ago

操作系统的详细版本，选择的CNI插件，kube-install版本是多少？ 从 k8s-master 到 k8s-node 的网络有什么限制吗？

CentOS Linux release 7.9.2009 (Core) kube-install-allinone-v0.8.0.tgz new centos7 systemctl stop firewalld

CYS158 commented 2 years ago

这个k8s-node是否执行了重复添加的动作？您可以再次尝试 delnode 和 addnode。

New system, first execution addnode

houseonline commented 2 years ago

Execute kubectl get cs,node -o wide on the command line to see whether the cluster information and node information are running normally??