[EKS] Support for Kubernetes v1.22

[tabern]

Tell us about your request EKS support for K8s v1.22

Which service(s) is this request for? Amazon EKS

[stevehipwell]

Could we get K8s v1.22 onto the Amazon EKS Kubernetes release calendar?

[mbevc1]

Any progress on this ticket? :thinking:

[stevehipwell]

I'm still waiting to see Kubernetes v1.22 on the release roadmap so we can start planning for this.

[617m4rc]

The lack of communication on this subject is a real headache. It is evident that 1.22 will require some extra work, but it would be assuring to know that this is being worked on already.

[johnkeates]

Well, this is a bummer. I thought a few months was the new release-to-EKS period to keep EKS more up-to-date but it appears that is not the case.

[xrl]

I can only assume Amazon is shifting priorities on EKS or the team is overwhelmed with outages and instability. Y'all hiring?

[johnkeates]

If EKS doesn't increase its predictability we can probably switch to rancher and just use the CNI, but I'd rather not.

[rubroboletus]

Dear Amazon, can you at least communicate? I know, that winter is comming, but I hope, you are not a bear.

[mreferre]

We are working on it, sorry for not having updated this issue sooner.

@xrl we think/hope we have been doing good with outages/instability but the EKS team is definitely growing and hence hiring. If you are interested in exploring the roles available please let me know offline. Thanks.

[stevehipwell]

Could we get K8s v1.22 onto the Amazon EKS Kubernetes release calendar?

@tabern @mreferre

[johnkeates]

Thanks for the response! 😀

[mreferre]

Could we get K8s v1.22 onto the Amazon EKS Kubernetes release calendar?

We are trying to be more cautious in how we communicate the availability of future releases because there are many factors that influence the cadence and there may be situations where a specific release takes longer to be available in the EKS service compared to our goals. Being more open while not creating false expectations is a fine line we all need to learn how to walk better. Needless to say we are always trying to improve our processes to be in a position to release new versions as soon as possible. I can say, however, that we will always trade "a few weeks" to match our high bar on security and stability. Which remain our priorities and what people came to love about (and expect from) EKS among other things.

[stevehipwell]

@mreferre I think a provisional date based on the current progress would be far better than leaving the calendar empty until shortly before the release. It's far better to be part of a conversation with a moving goal than to be left in the dark until the release date has been finalised. It'd be fine to asterisk uncertain release dates or even to use quarters, just as long as we're kept informed.

In this case we've only got until 2022-06-28 before Kubernetes v1.21 is official deprecated, so information enabling the planning of when we can do this work is really important.

[georgejohnis]

Hey everyone, sorry for the radio silence. Yes we are working on v1.22 for EKS. We expect this to be available in early 2022. We will update the EKS release calendar over the next few weeks as we narrow down on the exact date.

[inductor]

One thing that will be available(at least that I expect) is CSIMigration for Amazon EBS.

Right now I have running StatefulSets on my EKS cluster that uses in-tree storage provider with gp2 and I would like to migrate it to gp3 with EBS Storage Driver.

To do so, we need CSIMigration enabled as a feature gate on a cluster but at the moment EKS does not have it enabled, and it will be available as GA on 1.22.

Ref. https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/480

[stevehipwell]

@inductor did you see the Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes blog post?

[inductor]

@stevehipwell Oh I did not, I will have a look!

[617m4rc]

Given the delay in the rollout of 1.22, are there any plans to extend support for 1.21? The transition period until September 2022 is not exactly opulent.

[mikestef9]

Yes, we are extending 1.21 support to Feb '23, calendar will be updated soon.

[stevehipwell]

It looks like the calendar has been updated to show March 2022 as the release date for EKS v1.22.

[rubroboletus]

I understand, that releasing new version with all the vendor patches etc needs some time. But this long? At that time 1.24 will be almost released :-(

[stevehipwell]

@mikestef9 is there anything you can share with us around why the v1.22 release has taken so long? There may be things that the community could assist with moving forwards, but at a minimum it would help provide the context to understand and emphasise.

[pcgeek86]

kubectl is now complaining about minor version skew, because Amazon EKS is so far behind.

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

[johnkeates]

Yeah, this isn't great. I don't know if this is just an Amazon Thing™ at this point or if there is some complication with hosting a custom managed control plane, but I do know that the results just aren't there.

[chickenbeef]

kubectl is now complaining about minor version skew, because Amazon EKS is so far behind.

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

This is expected. You should use kubectl 1.20, 1.21 or 1.22: https://kubernetes.io/releases/version-skew-policy/#kubectl

[pcgeek86]

This is expected. You should use kubectl 1.20, 1.21 or 1.22:

I'm well aware of the version skew policy for Kubernetes. My point was simply that Amazon EKS is lagging far behind CNCF releases of Kubernetes.

[johnkeates]

I started using things like https://github.com/kenichi-shibata/kubectl-switch for EKS clusters because my other contexts are all higher versions and will complain about a version skew if I use a kubectl that is on EKS's level on those other apiservers...

[sftim]

This is my trick: https://www.scalefactory.com/blog/2019/10/16/using-multiple-kubectl-versions-with-minimal-faff/

[pcgeek86]

My strategy is to ignore the warning and just use the newest version of kubectl. 😆

[mikestef9]

Kubernetes v1.22 is a larger effort for us than many prior Kubernetes versions, as 1.22 removes a number of deprecated APIs, including older versions for webhooks, custom resources, certificate signing, and ingress. So in addition to control plane changes, we've been working on updating many other components that previously relied on deprecated API versions. For example, we are working on providing and documenting an official way to sign certificates, given 1.22 removes the "legacy-signer" functionality that many customers are using #750.

Additionally, behind the scenes we are moving to the out-of-tree AWS cloud provider as part of this launch, which allows us to move faster on AWS specific Kubernetes changes moving forward.

Appreciate the patience as we get closer to the launch target outlined in our version calendar. We also recognize v1.23 #1595 has been out for some time and v1.24 is expected in April, and we are considering multiple options for catching up.

[johnkeates]

Thanks again for the update @mikestef9, we're all very eager to see the upcoming release happen! It's also great to see a pretty precise date on the calendar!

[stevehipwell]

@mikestef9 are we still expecting EKS v1.22 to switch to Containerd by default? I think the original plan was to remove Docker altogether in v1.22, but it now looks like that will be happening in v1.23?

[mikestef9]

We delayed that change one version to 1.23. Dockerd will remain the default runtime in 1.22. See docs.

[stevehipwell]

We delayed that change one version to 1.23. Dockerd will remain the default runtime in 1.22. See docs.

I was aware of the removal being moved back to EKS v1.23 but I though the default change was still planned for EKS v1.22? The original announcement of removal in the same release as changing the default sounded bold and a bit of a stretch, do you think Docker will actually be removed from EKS v1.24 without Containerd having been the default for a cycle?

On a related note, https://github.com/awslabs/amazon-eks-ami/pull/855 would make it much easier to change bootstrap args such as container runtime on MNGs. I'd be happy to add some docs on it's use to the guide once it's merged.

[stevehipwell]

@mikestef9 has the release date moved? I'm pretty sure the calendar previously said March 17th but now it's just saying March?

[johnkeates]

I spotted that change as well. I guess they are close but not as sure as they were when the last calendar change was made. Maybe they'll even release earlier. But it's still march, we'll see it when we see it.

[pcgeek86]

Kubernetes version 1.24 is due out on April 19th.

https://www.kubernetes.dev/resources/release/

[inductor]

One good sign of this progress

https://github.com/bottlerocket-os/bottlerocket/pull/1962

[srenzo-perceptolab]

Any news on this topic? It's the end of March and still nothing on sight

[baileydoestech]

Why are AWS so slow on this?

Appreciate 1.22 is more 'involved' with a number of deprecations but looks like Azure and GKE are already on 1.23.

[rubroboletus]

Why are AWS so slow on this?

Appreciate 1.22 is more 'involved' with a number of deprecations but looks like Azure and GKE are already on 1.23.

they are just waiting for some special date, Maybe 1. of April

[tyrken]

Well, amazon-eks-node-1.22-v20220317 AMI is now visible in eu-west-1 ...

[pcgeek86]

Vultr has 1.23.5 available, as of March 23rd, 2022

https://www.vultr.com/zh/docs/vultr-kubernetes-engine-changelog/

Digital Ocean has offered 1.22 since March 3rd, 2022

https://docs.digitalocean.com/products/kubernetes/changelog/

[mariob1626]

At least the latest patch versions would be nice to have more often

[stevehipwell]

AKS has has v1.22 since December 9th 2021 and v1.23 has been in preview since February 16th 2022.

[inductor]

Everyone please calm down. I don't want any more notification that just complains about the schedule.

AWS has reasons to delay this and they put their schedule on it. Please hold on.

[pcgeek86]

@inductor Everyone is calm. People are politely stating facts about the Kubernetes ecosystem, and a few opinions. If you don't want notifications, you can click on the Unsubscribe button at the top of the issue page. GitHub offers this feature for anyone who wants to participate in a thread, but not get notifications about other comments.

AWS is a massive company with unimaginable amounts of revenue and profit. It's not unfair to have extremely basic expectations from them, such as keeping Kubernetes clusters up-to-date.

Have a blessed day.

[inductor]

@pcgeek86 I get it but at the same time it's not productive to keep saying the exact same thing about what is obviously not ignored and is recognised. I still need to be notified when 1.22 has officially been rolled out.

[stevehipwell]

I still need to be notified when 1.22 has officially been rolled out.

It usually hits the recent announcements blog and/or container blog first.

[Skaronator]

Totally agree with @inductor. I subscribed to this issue to get an email when stuff starts moving or even get released. Unnecessary comments here don't change anything. If you want to apply pressure to AWS, then use your AWS contact directly.

It usually hits the recent announcements blog and/or container blog first.

True, but you don't get an active notification, instead you would have to check the blog and filter by yourself. And you wouldn't update a production cluster within the first few days of release anyway, so subscribing to an issue is a valid use case.

---

Speaking of notifying about true progress. Just saw that the latest EKS AMI now contains a 1.22 version, which means that the EKS support is usually right around the corner (less than 10 days).

https://github.com/awslabs/amazon-eks-ami/releases/tag/v20220317