search

aws / eks-anywhere

Run Amazon EKS on your own infrastructure 🚀
https://anywhere.eks.amazonaws.com
Apache License 2.0
1.96k stars 283 forks source link

Upgrade of EKSA cluster on vSphere from 0.17.2 to 0.18.2 fails on patching vsphere-cloud-controller-manager. #7120

Closed jplewes closed 10 months ago

jplewes commented 10 months ago

Running into an odd issue which i cannot get past..

What happened: Upgrade of EKSA cluster on vSphere from 0.17.2 to 0.18.2 fails on patching vsphere-cloud-controller-manager.

message: 'patching object apps/v1, Kind=DaemonSet kube-system/vsphere-cloud-controller-manager: DaemonSet.apps "vsphere-cloud-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"k8s-app":"vsphere-cloud-controller-manager"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable'

What you expected to happen: Upgrade completes without failing.

How to reproduce it (as minimally and precisely as possible):
1) Install EKSA cluster with 0.17.2. and Kubernetes 1.24 on bottlerocket. 2) upgrade eks-anywhere tooling to 0.18.2 3) Edit cluster.yaml and increment kubernetes version to 1.25 4) run eksctl anywhere upgrade cluster -f cluster.yaml (at this point the cluster upgrades control plane nodes and etcd nodes, but doesn't upgrade any workers??) 5) edit cluster.yaml and change count of workers. 6) run eksctl anywhere upgrade cluster -f cluster.yaml (patching object error occurs)

Anything else we need to know?:
1) Bundled vSphere CSI was removed and v2.7.0 installed from vmware while on EKSA 0.17.2.
2) Cluster recently had etcd client certs expire.

jplewes commented 10 months ago

This issue can be closed. vSphere CPI had been replaced outside of EKSA, reverted and the upgrade proceeds.