Multi-node cluster upgrade "target version v1.xx.xx not supported" issue

What is version of KubeKey has the issue?

v3.0.7

What is your os environment?

v1.24.1 Ubuntu 20.04.4 LTS 5.4.0-136-generic containerd://1.6.4

KubeKey config file

root@lab-k8s-master-1:~# cat config-sample.yaml 
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: lab-k8s-master-1, address: 10.0.101.101, internalAddress: 10.0.101.101, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-master-2, address: 10.0.101.102, internalAddress: 10.0.101.102, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-master-3, address: 10.0.101.103, internalAddress: 10.0.101.103, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-node-1, address: 10.0.101.111, internalAddress: 10.0.101.111, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-node-2, address: 10.0.101.112, internalAddress: 10.0.101.112, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-node-3, address: 10.0.101.113, internalAddress: 10.0.101.113, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-node-4, address: 10.0.101.114, internalAddress: 10.0.101.114, user: ccieliu, password: "ccieliu"}
  - {name: lab-k8s-node-5, address: 10.0.101.115, internalAddress: 10.0.101.115, user: ccieliu, password: "ccieliu"}
  roleGroups:
    etcd:
    - lab-k8s-master-1
    - lab-k8s-master-2
    - lab-k8s-master-3
    control-plane: 
    - lab-k8s-master-1
    - lab-k8s-master-2
    - lab-k8s-master-3
    worker:
    - lab-k8s-node-1
    - lab-k8s-node-2
    - lab-k8s-node-3
    - lab-k8s-node-4
    - lab-k8s-node-5
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    internalLoadbalancer: haproxy
    domain: lb.cluster.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.24.13
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: none
    kubePodsCIDR: 100.64.64.0/18
    kubeServiceCIDR: 100.64.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []

A clear and concise description of what happend.

Hi Team,

Good day, that's a really amazing tool to help us create and upgrade kubernetes cluster. But as we check on testing cluster, some error occur in "upgrade" action.

ACTION TARGET: Direct upgrade multi-node k8s cluster from 1.24.1 >>> 1.24.13

WORK:

1.24.1 >>> 1.24.3
1.24.1 >>> 1.24.5
1.24.1 >>> 1.24.9

NOT WORK(Show release not supported):

1.24.1 >>> 1.24.13
1.24.1 >>> 1.24.10

Check "kubeadm upgrade plan", Output:


root@lab-k8s-master-1:~# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.24.1
[upgrade/versions] kubeadm version: v1.24.13
I0420 17:33:14.870002   44674 version.go:256] remote version is much newer: v1.27.1; falling back to: stable-1.24
[upgrade/versions] Target version: v1.24.13
[upgrade/versions] Latest version in the v1.24 series: v1.24.13

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 8 x v1.24.1 v1.24.13

Upgrade to the latest version in the v1.24 series:

COMPONENT CURRENT TARGET kube-apiserver v1.24.1 v1.24.13 kube-controller-manager v1.24.1 v1.24.13 kube-scheduler v1.24.1 v1.24.13 kube-proxy v1.24.1 v1.24.13 CoreDNS 1.8.6 v1.8.6

You can now apply the upgrade by executing the following command:

    kubeadm upgrade apply v1.24.13

The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no


- edit "kubekey config" from "v1.24.1"  to v1.24.13
- "kk upgrade -f config-sample.yaml --debug" output as following:

```sh
17:48:57 CST [UpgradeConfirmModule] Display confirmation form
+------------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| name             | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time         |
+------------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| lab-k8s-master-1 | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-master-2 | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-master-3 | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-node-1   | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-node-2   | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-node-3   | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-node-4   | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
| lab-k8s-node-5   | y    | y    | y       | y        | y     |       | y       | y         |        |        | v1.6.4     |            |             |                  | CST 17:48:54 |
+------------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

Cluster nodes status:
NAME               STATUS   ROLES           AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
lab-k8s-master-1   Ready    control-plane   307d   v1.24.1   10.0.101.101   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-master-2   Ready    control-plane   307d   v1.24.1   10.0.101.102   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-master-3   Ready    control-plane   307d   v1.24.1   10.0.101.103   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-node-1     Ready    worker          307d   v1.24.1   10.0.101.111   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-node-2     Ready    worker          307d   v1.24.1   10.0.101.112   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-node-3     Ready    worker          307d   v1.24.1   10.0.101.113   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-node-4     Ready    worker          307d   v1.24.1   10.0.101.114   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4
lab-k8s-node-5     Ready    worker          307d   v1.24.1   10.0.101.115   <none>        Ubuntu 20.04.4 LTS   5.4.0-136-generic   containerd://1.6.4

Upgrade Confirmation:
kubernetes version: v1.24.1 to v1.24.13

Continue upgrading cluster? [yes/no]: yes
17:49:11 CST success: [LocalHost]
17:49:11 CST [SetUpgradePlanModule 1/2] Set upgrade plan
17:49:11 CST success: [LocalHost]
17:49:11 CST [ProgressiveUpgradeModule 1/2] Calculate next upgrade version
17:49:11 CST message: [LocalHost]
calculate next version failed: the target version v1.24.13 is not supported
17:49:11 CST failed: [LocalHost]
error: Pipeline[UpgradeClusterPipeline] execute failed: Module[ProgressiveUpgradeModule 1/2] exec failed: 
failed: [LocalHost] [CalculateNextVersion] exec failed after 1 retires: calculate next version failed: the target version v1.24.13 is not supported

Relevant log output

Upgrade Confirmation:
kubernetes version: v1.24.1 to v1.24.13

Continue upgrading cluster? [yes/no]: yes
17:49:11 CST success: [LocalHost]
17:49:11 CST [SetUpgradePlanModule 1/2] Set upgrade plan
17:49:11 CST success: [LocalHost]
17:49:11 CST [ProgressiveUpgradeModule 1/2] Calculate next upgrade version
17:49:11 CST message: [LocalHost]
calculate next version failed: the target version v1.24.13 is not supported
17:49:11 CST failed: [LocalHost]
error: Pipeline[UpgradeClusterPipeline] execute failed: Module[ProgressiveUpgradeModule 1/2] exec failed: 
failed: [LocalHost] [CalculateNextVersion] exec failed after 1 retires: calculate next version failed: the target version v1.24.13 is not supported

Additional information

Question:

Please let me know if that's expected behavior?
Why "# kubeadm upgrade plan" check passed, but kubekey reject the action?
How to check kubekey "upgrade plan"? In other words, how to plan the upgrade path for kubernetes cluster which installed by "kubekey"?

Link function block https://github.com/kubesphere/kubekey/blob/49b32d3e8dcb696800536815a7db921e0907744f/pkg/kubernetes/tasks.go#L626

kubesphere / kubekey