failed to upgrade multi-node cluster from v2.1.1 to v3.0.0

what can i do to fix the problem? the logs as below:

[root@master ~]# ./kk upgrade --debug -f sample.yaml +--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+ | name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time | +--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+ | node4 | y | y | y | y | y | y | y | y | y | y | y | CST 12:48:34 | | node1 | y | y | y | y | y | y | y | y | y | y | y | CST 12:48:34 | | node2 | y | y | y | y | y | y | y | y | y | y | y | CST 12:48:34 | | node3 | y | y | y | y | y | y | y | y | y | y | y | CST 12:48:34 | | master | y | y | y | y | y | y | y | y | y | y | y | CST 12:48:35 | +--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+

Cluster nodes status: NAME STATUS ROLES AGE VERSION master Ready master 101d v1.16.7 node1 Ready worker 101d v1.16.7 node2 Ready worker 101d v1.16.7 node3 Ready worker 101d v1.16.7 node4 Ready worker 101d v1.16.7

Components Status: scheduler: ok controller-manager: ok etcd-0: {"health": "true"}

Upgrade Confirmation: kubernetes version: v1.16.7 to v1.18.6

Continue upgrading cluster? [yes/no]: yes INFO[12:48:59 CST] Get current version
INFO[12:49:04 CST] Configurating operating system ...
[master 192.168.2.38] MSG: net.ipv4.ip_forward = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 [node2 192.168.2.166] MSG: net.ipv4.ip_forward = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 vm.max_map_count = 655360 net.ipv4.ip_local_reserved_ports = 30000-32767 [node1 192.168.2.165] MSG: net.ipv4.ip_forward = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 vm.max_map_count = 655360 net.ipv4.ip_local_reserved_ports = 30000-32767 [node3 192.168.2.167] MSG: net.ipv4.ip_forward = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 vm.max_map_count = 655360 net.ipv4.ip_local_reserved_ports = 30000-32767 [node4 192.168.2.168] MSG: net.ipv4.ip_forward = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 vm.max_map_count = 655360 net.ipv4.ip_local_reserved_ports = 30000-32767 INFO[12:49:20 CST] Upgrading kube cluster
INFO[12:49:20 CST] Start Upgrade: v1.16.7 -> v1.17.9
INFO[12:49:20 CST] Downloading Installation Files
INFO[12:49:20 CST] Downloading kubeadm ...
INFO[12:49:21 CST] Downloading kubelet ...
INFO[12:49:22 CST] Downloading kubectl ...
INFO[12:49:22 CST] Downloading kubecni ...
INFO[12:49:23 CST] Downloading helm ...
[node4] Downloading image: kubesphere/pause:3.1 [node1] Downloading image: kubesphere/pause:3.1 [master] Downloading image: kubesphere/pause:3.1 [node2] Downloading image: kubesphere/etcd:v3.3.12 [node3] Downloading image: kubesphere/pause:3.1 [master] Downloading image: kubesphere/kube-apiserver:v1.17.9 [master] Downloading image: kubesphere/kube-controller-manager:v1.17.9 [master] Downloading image: kubesphere/kube-scheduler:v1.17.9 [master] Downloading image: kubesphere/kube-proxy:v1.17.9 [master] Downloading image: coredns/coredns:1.6.9 [master] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12 [node2] Downloading image: kubesphere/pause:3.1 [node4] Downloading image: kubesphere/kube-proxy:v1.17.9 [node1] Downloading image: kubesphere/kube-proxy:v1.17.9 [master] Downloading image: calico/kube-controllers:v3.15.1 [master] Downloading image: calico/cni:v3.15.1 [master] Downloading image: calico/node:v3.15.1 [node3] Downloading image: kubesphere/kube-proxy:v1.17.9 [master] Downloading image: calico/pod2daemon-flexvol:v3.15.1 [node2] Downloading image: kubesphere/kube-proxy:v1.17.9 [node1] Downloading image: coredns/coredns:1.6.9 [node4] Downloading image: coredns/coredns:1.6.9 [node3] Downloading image: coredns/coredns:1.6.9 [node4] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12 [node1] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12 [node2] Downloading image: coredns/coredns:1.6.9 [node3] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12 [node2] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12 [node4] Downloading image: calico/kube-controllers:v3.15.1 [node1] Downloading image: calico/kube-controllers:v3.15.1 [node3] Downloading image: calico/kube-controllers:v3.15.1 [node2] Downloading image: calico/kube-controllers:v3.15.1 [node4] Downloading image: calico/cni:v3.15.1 [node1] Downloading image: calico/cni:v3.15.1 [node3] Downloading image: calico/cni:v3.15.1 [node2] Downloading image: calico/cni:v3.15.1 [node4] Downloading image: calico/node:v3.15.1 [node1] Downloading image: calico/node:v3.15.1 [node3] Downloading image: calico/node:v3.15.1 [node2] Downloading image: calico/node:v3.15.1 [node1] Downloading image: calico/pod2daemon-flexvol:v3.15.1 [node4] Downloading image: calico/pod2daemon-flexvol:v3.15.1 [node2] Downloading image: calico/pod2daemon-flexvol:v3.15.1 [node3] Downloading image: calico/pod2daemon-flexvol:v3.15.1 INFO[12:52:38 CST] Upgrading master [192.168.2.38] node=192.168.2.38 Push /root/kubekey/v1.17.9/amd64/kubeadm to 192.168.2.38:/tmp/kubekey/kubeadm Done Push /root/kubekey/v1.17.9/amd64/kubelet to 192.168.2.38:/tmp/kubekey/kubelet Done Push /root/kubekey/v1.17.9/amd64/kubectl to 192.168.2.38:/tmp/kubekey/kubectl Done Push /root/kubekey/v1.17.9/amd64/helm to 192.168.2.38:/tmp/kubekey/helm Done Push /root/kubekey/v1.17.9/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.2.38:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done WARN[13:04:25 CST] Task failed ...
WARN[13:04:25 CST] error: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c "timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --allow-experimental-upgrades --allow-release-candidate-upgrades --etcd-upgrade=false --certificate-renewal=true --force" W0908 12:58:32.368749 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:32.369367 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:32.369401 27303 validation.go:28] Cannot validate kubelet config - no validator is available [upgrade/config] Making sure the configuration is correct: W0908 12:58:39.403103 27303 common.go:94] WARNING: Usage of the --config flag for reconfiguring the cluster during upgrade is not recommended! W0908 12:58:39.407041 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:39.407197 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:39.407211 27303 validation.go:28] Cannot validate kubelet config - no validator is available [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/version] You have chosen to change the cluster version to "v1.17.9" [upgrade/versions] Cluster version: v1.16.7 [upgrade/versions] kubeadm version: v1.17.9 [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler] [upgrade/prepull] Prepulling image for component kube-scheduler. [upgrade/prepull] Prepulling image for component kube-apiserver. [upgrade/prepull] Prepulling image for component kube-controller-manager. [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [upgrade/prepull] Prepulled image for component kube-scheduler. [upgrade/prepull] Prepulled image for component kube-controller-manager. [upgrade/prepull] Prepulled image for component kube-apiserver. [upgrade/prepull] Successfully prepulled the images for all the control plane components [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.17.9"... Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe Static pod: kube-controller-manager-master hash: 46dc50d3e6fde82fb8161e800378a02b Static pod: kube-scheduler-master hash: 751b9d6b14a9fd85d356850cc2b1dd14 [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests679436425" [controlplane] Adding extra host path mount "host-time" to "kube-controller-manager" W0908 12:59:17.214928 27303 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-08-12-59-17/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe [upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1 Error: Failed to upgrade kube cluster: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c "timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --allow-experimental-upgrades --allow-release-candidate-upgrades --etcd-upgrade=false --certificate-renewal=true --force" W0908 12:58:32.368749 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:32.369367 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:32.369401 27303 validation.go:28] Cannot validate kubelet config - no validator is available [upgrade/config] Making sure the configuration is correct: W0908 12:58:39.403103 27303 common.go:94] WARNING: Usage of the --config flag for reconfiguring the cluster during upgrade is not recommended! W0908 12:58:39.407041 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:39.407197 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:39.407211 27303 validation.go:28] Cannot validate kubelet config - no validator is available [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/version] You have chosen to change the cluster version to "v1.17.9" [upgrade/versions] Cluster version: v1.16.7 [upgrade/versions] kubeadm version: v1.17.9 [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler] [upgrade/prepull] Prepulling image for component kube-scheduler. [upgrade/prepull] Prepulling image for component kube-apiserver. [upgrade/prepull] Prepulling image for component kube-controller-manager. [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [upgrade/prepull] Prepulled image for component kube-scheduler. [upgrade/prepull] Prepulled image for component kube-controller-manager. [upgrade/prepull] Prepulled image for component kube-apiserver. [upgrade/prepull] Successfully prepulled the images for all the control plane components [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.17.9"... Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe Static pod: kube-controller-manager-master hash: 46dc50d3e6fde82fb8161e800378a02b Static pod: kube-scheduler-master hash: 751b9d6b14a9fd85d356850cc2b1dd14 [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests679436425" [controlplane] Adding extra host path mount "host-time" to "kube-controller-manager" W0908 12:59:17.214928 27303 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-08-12-59-17/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe [upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1 Usage: kk upgrade [flags]

Flags: -f, --filename string Path to a configuration file -h, --help help for upgrade --skip-pull-images Skip pre pull images --with-kubernetes string Specify a supported version of kubernetes --with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)

Global Flags: --debug Print detailed information (default true)

Failed to upgrade kube cluster: Failed to upgrade master: master: Failed to exec command: sudo -E /bin/sh -c "timeout -k 600s 600s /usr/local/bin/kubeadm upgrade apply -y v1.17.9 --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --allow-experimental-upgrades --allow-release-candidate-upgrades --etcd-upgrade=false --certificate-renewal=true --force" W0908 12:58:32.368749 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:32.369367 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:32.369401 27303 validation.go:28] Cannot validate kubelet config - no validator is available [upgrade/config] Making sure the configuration is correct: W0908 12:58:39.403103 27303 common.go:94] WARNING: Usage of the --config flag for reconfiguring the cluster during upgrade is not recommended! W0908 12:58:39.407041 27303 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] W0908 12:58:39.407197 27303 validation.go:28] Cannot validate kube-proxy config - no validator is available W0908 12:58:39.407211 27303 validation.go:28] Cannot validate kubelet config - no validator is available [preflight] Running pre-flight checks. [upgrade] Making sure the cluster is healthy: [upgrade/version] You have chosen to change the cluster version to "v1.17.9" [upgrade/versions] Cluster version: v1.16.7 [upgrade/versions] kubeadm version: v1.17.9 [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler] [upgrade/prepull] Prepulling image for component kube-scheduler. [upgrade/prepull] Prepulling image for component kube-apiserver. [upgrade/prepull] Prepulling image for component kube-controller-manager. [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager [upgrade/prepull] Prepulled image for component kube-scheduler. [upgrade/prepull] Prepulled image for component kube-controller-manager. [upgrade/prepull] Prepulled image for component kube-apiserver. [upgrade/prepull] Successfully prepulled the images for all the control plane components [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.17.9"... Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe Static pod: kube-controller-manager-master hash: 46dc50d3e6fde82fb8161e800378a02b Static pod: kube-scheduler-master hash: 751b9d6b14a9fd85d356850cc2b1dd14 [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests679436425" [controlplane] Adding extra host path mount "host-time" to "kube-controller-manager" W0908 12:59:17.214928 27303 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-09-08-12-59-17/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-master hash: fe913818f3dca703aafb0c727918bfbe [upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition To see the stack trace of this error execute with --v=5 or higher: Process exited with

kubesphere / kubekey

failed to upgrade multi-node cluster from v2.1.1 to v3.0.0 #290