Open Gier32o opened 2 months ago
Hello @Gier32o, does /var/log/aws-routed-eni/plugin.log
or /var/log/aws-routed-eni/ipamd.log
logs show any detailed about on the ip assignment or failure? Is aws-node pod running?
Usually during K8s upgrade, the CNI version does not change, we keep the CNI version same while performing K8s upgrade. After the k8s upgrade, you can do the CNI upgrade. Does this workflow give the desirable outcome?
Hi, the aws-node pods are running fine. You were right - upgrading addon version before or at the same time as kubernetes and worker AMIs results in this error. If I run upgrade in two batches: 1. (K8s + AMIs) -> 2. (Addon) it works fine. Thanks! Is there any way to fix such a broken cluster afterwards?
Is there any way to fix such a broken cluster afterwards?
I am not sure what could have led to this stage. But you can downgrade the addon the previous version, and restart the pods, and upgrade the addons again.
Hi @orsenthil, I upgraded in the order you recommend, (K8s + AMIs) first, then Addon, but got the same problem. It even randomly failed, not all the time.
vpc_cni_version = "v1.16.3-eksbuild.2"
kubernetes_version = "1.28"
to
vpc_cni_version = "v1.18.2-eksbuild."
kubernetes_version = "1.29"
Error: updating EKS Add-On (test:vpc-cni): operation error EKS: UpdateAddon, https response error StatusCode: 400, RequestID: 608f24fe-795a-4c7c-acba-8d11836aa01b, InvalidParameterException: Addon version specified is not supported
when trying to downgrade the plugin 1.18.2
-> 1.16.3
.
Nothing changed when I downgraded to 1.17.1
So is it a bug or is it something wrong with configuration?
Pods are stuck in ContainerCreating status after upgrading to Kubernetes version 1.30 on EKS. We have 'Security Groups for Pods' feature turned on, and when we're trying to upgrade from:
to
we're getting failing pods:
There is no issue when 'Security Groups for Pods' feature is turned off How to reproduce: https://github.com/Gier32o/k8s-upgrade-problem