tanzu cluster upgrade to incompatible version changes label "tanzuKubernetesRelease"

[chrischdi]

Bug description

• Did create a cluster using v1.20
• Did run tanzu cluster upgrade <cluster-name> (tanzu cli v1.5.4)
• cli tried to upgrade v1.20 cluster to incompatible latest kubernetes version v1.22 which failed (as expected).
• while trying to upgrade the cli changes the cluster label at .metadata.labels.tanzuKubernetesRelease to the target version xref
• this change does not get reverted after the failure

After the failed upgrade it is not possible to upgrade using a compatible version. even tanzu cluster available-upgrades get <cluster-name> does not show a compatible version to upgrade to anymore.

Affected product area (please put an X in all that apply)

• [ ] APIs
• [ ] Addons
• [x] CLI
• [ ] Docs
• [ ] IAM
• [ ] Installation
• [ ] Plugin
• [ ] Security
• [ ] Test and Release
• [ ] User Experience

Expected behavior

CLI does not even start the upgrade (and does not change the label) when using an incompatible version to upgrade to.

Or at least reverts the label change for some failures (e.g. when upgrade did not even start).

Steps to reproduce the bug

• Create v1.20 cluster
• Try tanzu cluster upgrade to upgrade the cluster to v1.22+

Version (include the SHA if the version is not obvious)

TKG v1.5.4

ubuntu@ip-172-31-9-29:~$ tanzu version
version: v0.11.6
buildDate: 2022-05-20
sha: 90440e2b
ubuntu@ip-172-31-9-29:~$ k get providers -A
NAMESPACE                           NAME                    TYPE                     PROVIDER      VERSION   WATCH NAMESPACE
capa-system                         infrastructure-aws      InfrastructureProvider   aws           v1.2.0
capi-kubeadm-bootstrap-system       bootstrap-kubeadm       BootstrapProvider        kubeadm       v1.0.1
capi-kubeadm-control-plane-system   control-plane-kubeadm   ControlPlaneProvider     kubeadm       v1.0.1
capi-system                         cluster-api             CoreProvider             cluster-api   v1.0.1

Environment where the bug was observed (cloud, OS, etc)

AWS

Relevant Debug Output (Logs, manifests, etc)

ubuntu@ip-172-31-9-29:~$ tanzu cluster upgrade start-1-20
Upgrading workload cluster 'start-1-20' to kubernetes version 'v1.22.9+vmware.1'. Are you sure? [y/N]: y
Validating configuration...
updating additional components: 'addons-management/kapp-controller' ...
Verifying kubernetes version...
Error: kubernetes version verification failed: Upgrading Kubernetes from v1.20.14+vmware.1 to v1.22.9+vmware.1 is not supported
...

[github-actions[bot]]

Hey @chrischdi! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Tanzu Framework.

[vuil]

@anujc25 can you PTAL, thanks.