Upgrade to Kubernetes 1.22

[smerle33]

Previous upgrade: https://github.com/jenkins-infra/helpdesk/issues/2866

• [x] Upgrade kubectl within docker-helmfile

• [x] Upgrade DOKS

  • [x] Ensure cluster is re-created by terraform in jenkins-infra/digitalocean
    • Initial Kubernetes 1.22 upgrade digitalocean/variables.tf https://github.com/jenkins-infra/digitalocean/pull/43
  • [x] Add admin's kubeconfig in the infra.ci's SOPS credentials: (private repo)
  • [x] Ensure that cluster is still off and not to be used by ci.jenkins.io while no more credit to use are available
  • [x] Add the credential of the service account jenkins-agents in ci.jenkins.io (manual step)

• [x] Upgrade EKS

  • [x] Open datetime announcement on status.jenkins.io (next time)
  • [x] Disable cluster in ci.jenkins.io's config as code
  • need to remove the whole block (0 as max-capacity is not stopping agent to be spawn) https://github.com/jenkins-infra/jenkins-infra/blob/production/hieradata/clients/azure.ci.jenkins.io.yaml#L41-L130
  • [x] Upgrade cluster through Terraform (control plane + node groups)
  • [x] Upgrade add-ons (weird: their version are not upgraded by the EKS module. Need to be specified as terraform variable + updatecli-zed from the tables at https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html, https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html and https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)
  • if done by UI, select the checkbox "replace existing configuration"
  • and for vpc-cni select the inheritance from node.
  • [x] Ensure that upgraded cluster is still ok (check kubernetes-management)
  • [x] Enable it again in ci.jenkins.io config as code

• [x] Upgrade AKS

  • [x] Check previous AKS upgrades and create one for 1.22
  • [x] Open datetime announcement on status.jenkins.io
  • [x] Upgrade temp-privatek8s first. ⚠️ it will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job
  • [x] Upgrade prodpublick8s
  • [x] Close announcement

[github-actions[bot]]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% #2866
2. 77% #2664

[dduportal]

• Initial kubectl 1.22 PR: https://github.com/jenkins-infra/docker-helmfile/pull/105, replaced by https://github.com/jenkins-infra/docker-helmfile/pull/116
  • Was blocked until today by https://github.com/jenkins-infra/kubernetes-management/pull/2403 but it's now UNBLOCKED.
  • Ping @smerle33 , ready for you to continue on kubectl upgrade in https://github.com/jenkins-infra/docker-helmfile/pull/116

[dduportal]

Ping @smerle33 : as https://github.com/jenkins-infra/digitalocean/pull/43 is merged, it looks to me that DOKS is done?

[smerle33]

from ci.jenkins.io

[lemeurherve]

from ci.jenkins.io

... for the Digital Ocean cluster.

[dduportal]

👏

[smerle33]

upgrade EKS : ~ version = "1.21" -> "1.22"

[smerle33]

manual upgrade of coredns : 1.8.4 --> 1.8.7-eksbuild.1 manual upgrade of kube-proxy : 1.21.2-eksbuild.2 --> 1.22.6-eksbuild.1 manual upgrade of vpc-cni : v1.10.3-eksbuild.1 --> v1.11-eksbuild.1

[lemeurherve]

Selected notes from changelog for AKS 1.22:

release-2022-02-24

Behavioral changes The default VNET address for managed VNETs will change from 10.0.0.0/8 to 10.240.0.0/16 and the default node subnet address will change from 10.224.0.0/12 to 10.224.0.0/16. New clusters will be required to have service and pod CIDR ranges that do not overlap with these new VNET ranges. Bug fixes Fix azure disk resize timeout issue on aks 1.21+

release-2022-03-27

Features Node pool Scale-down Mode is not Generally available and supports Spot Node Pools.

release-2022-05-01

Public preview

• The aks-preview Azure CLI extension (version 0.5.66+) now supports running az aks update -g -n without any optional arguments. This will perform an update operation without performing any changes, which can recover a cluster stuck in a failed provisioning state.
• AKS now supports updating kubelet on node pools to use a new or changed user-assigned managed identity.

release-2022-05-22

Features

• Azure CNI dynamic IP allocation and enhanced subnet support is now generally available.
• Alias minor version is now generally available.

  As an Azure Kubernetes Service (AKS) user, you must specify the exact patch number in your setup. With the Kubernetes version alias feature release, we are simplifying setup for you so that you don’t need to specify patch number. For example, you can specify Kubernetes 1.20 instead of Kubernetes 1.20.1. By not specifying the exact patch number, you will automatically be placed on the highest patch version of the minor version you selected.

Preview features

• ARM64 agent pools is now in public preview.
• Windows Server 2022 host support is now in public preview.

release-2022-06-12

Behavioral Changes Upgrades spot node pools is now available starting this week: When upgrading a spot node pool, AKS will issue a cordon and an eviction notice, but no drain is applied. There are no surge nodes available for spot node pool upgrades.

release-2022-06-19

Announcements Kubernetes 1.21 version deprecation will start taking effect from July 31st, 2022.

Selected note from changelog for AKS 1.23:

release-2022-01-07

Starting with 1.23 AKS will follow upstream kubernetes and deprecate in-tree azure authentication which is marked for deprecation to be replaced with 'exec'. If you are using Azure CLI or Azure clients, AKS will download kubelogin for users automatically. If outside of Azure CLI, users need to download and install kubelogin in order to continue to use kubectl with AAD authentication. https://github.com/Azure/kubelogin

Selected note from changelog for AKS 1.24:

release-2022-06-12

Starting with Kubernetes 1.24, the following changes will be made:

• The default format of clusterUser credential for AAD enabled clusters will be ‘exec’, which requires kubelogin binary in the execution PATH. If you are using Azure CLI, it will prompt users to download kubelogin. There will be no behavior change for non-AAD clusters, or AAD clusters whose version is older than 1.24. Existing downloaded kubeconfig will still work. We provide an optional query parameter ‘format’ when getting clusterUser credential to overwrite the default behavior change, you can explicitly specify format to ‘azure’ to get old format kubeconfig.
• CoreDNS version 1.9.2 will be default version. With this new version of CoreDNS wildcard queries are no longer allowed.

[dduportal]

Status discussed during today's weekly meeting:

• AKS clusterS planned thursday 30th of June starting at 07:00UTC (communications sent)
• Once all clusters are upgraded, you'll have to write up the issue "Upgrade to Kubernetes 1.23" issue with ALL the things that:
  • Were forgotten in the "todo list"
  • Went wrong
  • you detected to be on the 1.23 changelog

Once these 2 tasks are ok, we can close the issue :)

[daniel-beck]

Noting that LDAP and get.jenkins.io are currently unavailable due to problems.

[lemeurherve]

We’ve identified the problem and working on it, will post an update as soon as it's resolved.

[lemeurherve]

LDAP restored, next: get.jenkins.io (same kind of problem)

[lemeurherve]

get.jenkins.io restored too

[lemeurherve]

Upgrade to Kubernetes 1.22 completed, sorry for the longer outage than expected, preparing a short post-mortem.

[dduportal]

One last step: create the issue in jenkins-infra/helpdesk for 1.23 from the current one + add the improvements/changes in it, so it should be a no brainer next time :)

[lemeurherve]

Issue created.