Upgrade to Kubernetes 1.26

[dduportal]

Previous upgrade (1.25): https://github.com/jenkins-infra/helpdesk/issues/3582

Depreciation timelines for 1.25 (justifying the upgrade to 1.26):

• Digital Ocean: Oct. 2023 - https://docs.digitalocean.com/products/kubernetes/details/supported-releases/
• Azure AKS: Dec. 2023 - https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar
• AWS EKS: May 2024 - https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar

---

Task list:

• [x] Upgrade kubectl within docker-helmfile

  • [x] First upgrade the updatecli manifest
  • [x] Then apply the updatecli PR that is produced by this change:
  • [x] Then Apply the Kubernetes-Management PR that will use this new version -

• [x] Upgrade DOKS (doks and doks-public) - see https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1780857936 below

• [x] Upgrade AWS EKS (cik8s and eks-public) - see https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1794913444

• [x] Upgrade AKS (privatek8s and publick8s) - https://github.com/jenkins-infra/helpdesk/issues/3683#issuecomment-1797928819

[github-actions[bot]]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% #3387
2. 92% #3053
3. 92% #2930
4. 92% #2866
5. 77% #2664

[lemeurherve]

To keep in mind: https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

To ensure the separation between the open source version of Kubernetes and those versions that are customized by services providers [...], the open source community is requiring that all provider-specific code that currently exists in the OSS code base be removed starting with v1.26.

[smerle33]

kubectl 1.26.9 is now used in our system (client side).

[dduportal]

Update:

• WiP on DigitalOcean clusters:
  • Changelog for Kube 1.26 being checked with a focus on https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#urgent-upgrade-notes
  • No breaking changes for DO and we don't use directly the deprecated APIs (https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-26). Eventually some of the 3rd-party helm-chart we use do it, but the upgrade and an helmfile apply will fix it (if not already done with 1.25 server-side and 1.26 client-side)
  • DO has their own changelog for their managed components: https://docs.digitalocean.com/products/kubernetes/details/changelog/#1-26-x
  • All components are upgraded as expected, no major change catching attention (but it is managed so nothing to do really)
  • Note: Debian on the nodes will be upgraded featuring a kernel 6.x
  • Preparing upgrade of the cluster, the 2 at the same time (doks-public and doks)
  • Requires disabling DO from agent clouds in ci.jenkins.io at least 15 min before the operation
  • Need to announce operation (pick a date time, and then usual status/mailing/IRC, etc.)

TODO:

• EKS
• AKS

[dduportal]

Update on DigitalOcean upgrade:

• Announcement: Thursday 26 Oct. at noon UTC
  • [x] status.jenkins.io - https://github.com/jenkins-infra/status/pull/396
  • [x] developer mailing list (as ci.jenkins.io need DOKS to be disabled for agents) + jenkins-infra mailing - https://groups.google.com/g/jenkinsci-dev/c/dW_SzUO_iv0
  • [x] IRC #jenkins-infra - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$ITwVciwfKxpadSW_6Q5MwAiwdwqfXvLX-wnXje0HywA?via=libera.chat&via=gitter.im&via=matrix.org
  • [x] Add a banner on ci.jenkins.io
• [x] Disable DO Kubernetes Cloud agent in ci.jenkins.io (puppet hieradata PR) - https://github.com/jenkins-infra/jenkins-infra/pull/3140
• [x] Disable kubernetes-management of both clusters (PR) - https://github.com/jenkins-infra/kubernetes-management/pull/4588
• [x] Bump Kube version to 1.26 for doks-public
  • [x] Terraform PR - https://github.com/jenkins-infra/digitalocean/pull/168
  • [x] Upgrade triggered manually through the DO portal (see comment below): Your cluster has successfully upgraded to version 1.26.9-do.0.
  • [x] Sanity check with a local helmfile diff (no change)
• [x] Bump Kube version to 1.26 for doks
  • [x] Terraform PR - https://github.com/jenkins-infra/digitalocean/pull/167
  • [x] Upgrade triggered manually through the DO portal (see comment below): Your cluster has successfully upgraded to version 1.26.9-do.0.
  • [x] Sanity check with a local helmfile diff (no change)
• [x] Terraform jenkins-infra/digitalocean project has a successfull build without changes on the main branch after the upgrades
  • Had to disable the job, merge the 2 upgrade PR, run the manual upgrade, check locally and re-enable build
  • https://infra.ci.jenkins.io/job/terraform-jobs/job/digitalocean/job/main/528 => No changes. Your infrastructure matches the configuration.
• [x] Enable kubernetes-management of both clusters (PR) - https://github.com/jenkins-infra/kubernetes-management/pull/4589
  • PR had no helmfile diff changed advertised (map with the manual sanity checks ran earlier) ✅
  • Main branch is successfull: https://infra.ci.jenkins.io/job/kubernetes-jobs/job/kubernetes-management/job/main/24962 ran with success, no changes and now warnings!
• [x] Enable DO Kubernetes Cloud agent in ci.jenkins.io (puppet hieradata PR) - https://github.com/jenkins-infra/jenkins-infra/pull/3141
  • Upgrade to Kubernetes 1.26 went well (no issue, no warning)
  • The ACP service in DigitalOcean is up and running (https://repo.do.jenkins.io/public/org/jenkins-ci/plugins/plugin/4.61/)
• [x] Validate agents can be spawned using a replay of the Infra/acceptance-tests and a simple scripted pipeline node('doks && maven') { sh 'mvn -v' }
  • https://ci.jenkins.io/job/Infra/job/acceptance-tests/job/check-agent-availability/3016 was successful 👍
• [x] Close announcements

Post Mortem

The upgrade itself went fine but it had to be done through the DigitalOcean UI: the 2 terraform PRs did not show any change in their plans because of https://github.com/jenkins-infra/digitalocean/pull/148 which tells Terraform to ignore the version changes.

Rollbacking this change fails any terraform plan (as explained in the PR) due to the way how digitalocean_kubernetes_cluster and data.digitalocean_kubernetes_cluster are linked in relation with the kuberbetes providers in charge of managing CSI and admin SVCaccounts.

As such, the upgrade procedure is amended to the following workflow:

• Keep the terraform PRs (one or 2 clusters at a time, both can be done)
• Disable the Terraform Job
• Run the upgrades through the Digital Ocean Web portal
• Merge the PRs without the checks
• Run a terraform plan from upstream/main as a sanity check to ensure to changes or cluster destruction are planned
• Enable the Job and run it once to ensure its is successfull and up to date with upgraded clusters
• Enable Terraform

[dduportal]

EKS changelogs:

• Reminder: Upgrade note from 1.25 to 1.26 for Kubernetes itself
• EKS 1.26 Release Notes
• A testimony of a Terraform-managed EKS upgrade from 1.25 to 1.26

TL;DR:

• We have to check for the VPC CNI plugins to be on version 1.12 or later before the upgrade
• No problem on the containerd version (removed CRI v1alpha2) as we are using Amazon default AMI images
• With 1.26, EKS is finally having the same feature set for CSI fsGroup option. We don't really care as only our ACP services uses persistent volume in EKS, but worth having in mind

[dduportal]

Update on AWS EKS upgrade:

• Announcement: Monday 06 Nov. at 14h00 UTC
  • [x] status.jenkins.io - https://github.com/jenkins-infra/status/pull/408
  • [x] developer mailing list (as ci.jenkins.io need DOKS to be disabled for agents) + jenkins-infra mailing - https://groups.google.com/g/jenkinsci-dev/c/2baX6c7oFzY
  • [x] IRC #jenkins-infra - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$Q5usI9ioxXMm1YT0Gl-chiwpAJ-dURyhNVxMywQextU?via=libera.chat&via=gitter.im&via=matrix.org
  • [x] Add a banner on ci.jenkins.io
• [x] Ensure the VPC CNI plugins to be on version 1.12 or later on:
  • [x] eks-public - https://github.com/jenkins-infra/aws/blob/632f2f3166f79eb80793889250d7abeed5f3a75a/eks-public-cluster.tf#L67
  • [x] cik8s - https://github.com/jenkins-infra/aws/blob/632f2f3166f79eb80793889250d7abeed5f3a75a/cik8s-cluster.tf#L78
• [x] Disable AWS/BOM Kubernetes Clouds in ci.jenkins.io (puppet hieradata PR) - https://github.com/jenkins-infra/jenkins-infra/pull/3169
• [x] Disable kubernetes-management of both clusters (PR) - https://github.com/jenkins-infra/kubernetes-management/pull/4635
• [x] Bump Kube version to 1.26 for cik8s
  • [x] Terraform PR with both Kubernetes and addons update - https://github.com/jenkins-infra/aws/pull/482
  • [x] Sanity check with a local helmfile diff (no change)
• [x] Bump Kube version to 1.26 for eks-public
  • [x] Terraform PR with both Kubernetes and addons update - https://github.com/jenkins-infra/aws/pull/483
  • [x] Sanity check with a local helmfile diff (no change)
• [x] Terraform jenkins-infra/aws project has a successful build without changes on the main branch after the upgrades - https://infra.ci.jenkins.io/job/terraform-jobs/job/aws/job/main/436/
• [x] Enable kubernetes-management of both clusters (PR) - https://github.com/jenkins-infra/kubernetes-management/pull/4636
• [x] Check the ACP service in AWS is up and running (https://repo.aws.jenkins.io/public/org/jenkins-ci/plugins/plugin/4.61/)
• [x] Enable AWS/BOM Kubernetes Clouds in ci.jenkins.io (puppet hieradata PR) - https://github.com/jenkins-infra/jenkins-infra/pull/3170
• [x] Validate agents can be spawned using a replay of the Infra/acceptance-tests and a simple scripted pipeline node('cik8s && maven') { sh 'mvn -v' } and node('bom && maven') { sh 'mvn -v' }
• [x] Close announcements

Post Mortem

• We had an issue with the addon VPC-CNI (only this one) which cannot be upgraded more than one minor increment at a time. Since it was 1.13, we had to upgrade manually (for both clusters) to 1.14.1, and then we've let Terraform perform the 1.14.1 -> 1.15.x upgrade.
  • TODO: we need to implement https://github.com/jenkins-infra/aws/issues/278 for VPC-CNI or plan to upgrade addons manually prior to the next EKS upgrade.

[dduportal]

Update: AKS Upgrade plan

• [x] Check changelog:

Current changelog notable elements for Kubernetes 1.26:

• Some AKS labels are being deprecated with the Kubernetes 1.26 release. Update your AKS labels to the recommended substitutions. See more information on label deprecations and how to update your labels in the Use labels in an AKS cluster documentation. beta.kubernetes.io/arch= and beta.kubernetes.io/os= are still applied by kubelet in kubernetes code
  • ✅ We don't seem to use these labels since 1 year at least (no warning from helmfiles)
• HostProcess Containers will be GA
  • ✅ We don't use this feature except when debugging
• Two in-tree driver persistent volumes won't be supported in AKS : kubernetes.io/azure-disk, kubernetes.io/azure-file.
  • ✅ All of our PV are using CSI already
• 📝 (for info) All AKS clusters on version 1.26+ will use the latest coreDNS version v1.10.1. (see below)
  • For all AKS clusters on version 1.26+, coreDNS health plugin will use lameduck 5s to minimizes DNS resolution failures during coreDNS pod restart or deployment rollout.
  • For all AKS clusters on version 1.26+, coreDNS will use ttl 30 as default TTL for DNS records.
• During cluster upgrade to v1.26.0 or a later version, disk PV node affinity check will cause the upgrade to fail if there are disk PVs still using deprecated labels: failure-domain.beta.kubernetes.io/zone and failure-domain.beta.kubernetes.io/region
  • ✅ All of our PV are using CSI already
  • ✅ Temp. fix by m$: Enable failure-domain.beta.kubernetes.io labels on K8S 1.26+ nodes by default to resolve issue with in tree CSI drivers. Will be removed from K8S 1.28

privatek8s

• Announcement: Tuesday 07 Nov. 2023 at 08:00 UTC
  • [x] Open datetime announcement on status.jenkins.io - https://github.com/jenkins-infra/status/pull/413
  • [x] jenkins-infra mailing - https://groups.google.com/g/jenkins-infra/c/x3sh2Vsveb0
  • [x] IRC #jenkins-infra - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$Gk1w1qENvCc7VKKoqwpXo0Ah4VcHAgBeeigsjkAIAQ4?via=libera.chat&via=gitter.im&via=matrix.org
• [x] Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - https://github.com/jenkins-infra/kubernetes-management/pull/4640
• [x] Terraform PR to upgrade cluster to 1.26 but NO merge - https://github.com/jenkins-infra/azure/pull/505
• [x] Disable terraform azure job (⚠️ As the upgrade will put infra.ci out of service so need to be run manually)
• [x] Make sure local (on your machine) repository is up to date and you are logged-in: run a terraform plan
• [x] Merge Terraform PR even without checks (as infra.ci restart will re-enable job)
• [x] Perform upgrade from local machine
• [x] Check that helmfile diff still works and shows no change and no warning
• [x] Wait for infra.ci to be up again
• [x] Enable kubernetes management for privatek8s cluster and check it runs successfully - https://github.com/jenkins-infra/kubernetes-management/pull/4642
• [x] Check terraform job and eventually trigger it for success with no changes
• [x] Check for release.ci to be up and running: try to allocate agents on release.ci with a job replay (dummy linux and a dummy window)

Post Mortem for privatek8s

• infra.ci did not restart due to a JCasc syntax error. Hot-fixed by https://github.com/jenkins-infra/kubernetes-management/pull/4641 with context
• The cluster upgrade was refused by Azure API (both Terraform and Web console failed) due to a lock placed on the MC_... resource group in https://github.com/jenkins-infra/helpdesk/issues/3582#issuecomment-1629210833 (last Kubernetes upgrade).
  • 🔧 We had to delete manually the lock before performing the upgrade
  • ✅ No Public IP were broken during the operation (good to know for publick8s...)
  • ⚠️ Might be worth applying https://github.com/jenkins-infra/helpdesk/issues/3582#issuecomment-1629752851 as part of the publick8s operation

[dduportal]

Update:

• Migration tests of the Public IPs on privatek8s
  • Manual tests:
  • The resource group MC_prod-privatek8s_privatek8s-emerging-ram_eastus2 (managed by the privatek8s) has 2 objects of type Public IP used by the cluster:
    • 1 is used for inbound requests, is terraform-managed and named public-privatek8s
    • 1 is used for outbound requests, is AKS-managed and has a generated unique name
  • ✅ First manual test:
    • ✅ We created manually a new RG named prod-public-ips
    • ✅ Through the Azure Web UI, we moved the outbound IP from MC_prod-privatek8s_privatek8s-emerging-ram_eastus2 to this new RG
    • Took ~5 min
    • No outbound connectivity error during the migration
    • ⚠️ 10 min after the migration, we started to see errors in infra.ci.jenkins.io when reaching the publick8s control plane 🤔
    • The Public IP being managed by the cluster, it created a new one transparently, which effectively changed the outbound IP
    • 4 hotfixes to update this IP:
      • https://github.com/jenkins-infra/azure/commit/25df2e5310630554a267d41e2f4a19e6a5b56d61
      • https://github.com/jenkins-infra/terraform-states/commit/75644bf0bd44f3cce7af0edc775677f82f166844
      • https://github.com/jenkins-infra/aws/commit/8ed82cb4db9c0719ea6d434ed09c77381b601a03
      • https://github.com/jenkins-infra/kubernetes-management/commit/c6e330e3bbe85cadfbfaaa3ea696e339af35fb5f
  • ✅ Second test (half manual / half automated)
    • ✅ With jenkins-infra/azure and jenkins-infra/kubernetes-management djobs disabled (mandatory!)
    • ✅ Through the Azure Web UI, we moved the inbound IP from MC_prod-privatek8s_privatek8s-emerging-ram_eastus2 to this new RG
    • Took ~5 min
    • No inbound connectivity error during the migration (tested by redelivering webhooks payloads from GitHub to infra.ci)
    • ✅ We updated the SVC LB annotation (kubectl -n public-nginx-ingress edit svc public-nginx-ingress-ingress-nginx-controller) to:
    • Documentation from @timja comment which leads to https://learn.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address
    • Removed the service.beta.kubernetes.io/azure-load-balancer-ipv4 annotation in favor of service.beta.kubernetes.io/azure-pip-name as recommended by the documentation (introduced way after our initial IPv6 implementation)
    • Added the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Update successfull and no inbound connectivity error
    • ⚠️ The SVC LB used for the public ingress of privatek8s cluster showed warning messages in its events (kubectl -n public-nginx-ingress describe svc public-nginx-ingress-ingress-nginx-controller) about missing permissions .../join on the Public IP
    • We need to add a new role assignement in Azure AD (Entra) as the moved Public IP needs "Network Contributor" role's permissions now it's not dependending on the AKS-managed RG
    • ✅ Put Azure changes in IaC (Terraform) on jenkins-infra/azure:
    • Locally: imported the new RG prod-public-ips, updated the public IP public-privatek8s + its lock resource and adding the ne role assignement
    • PR merged without waiting for checks, applied manually and verified by enabling the jenkins-infra/azure jobs
    • See https://github.com/jenkins-infra/azure/pull/507
    • Note the checks failed due to the public IP changes mentionned earlier, which we fixed afterwards
    • ✅ Put Kubernetes changes in IaC on jenkins-infra/kubernetes-management:
    • PR applied manuually (locally) during the operation, and then merged without waiting for checks
    • https://github.com/jenkins-infra/kubernetes-management/pull/4645
    • Note the checks failed due to the public IP changes mentionned earlier, which we fixed afterwards
    • ⚠️ We had to wait ~ 10 min to see the warning messages disapearring on the SVC LB (time for the permissions to propagate)

TODO:

• Prepare plan for the "Public IP migration" for publick8
• Announce operation on publick8s which will consist in:
  • Public IP migration
  • Falco upgrade on publick8s to ensure it works on arm64
  • Nginx ingress upgrade on publick8s to ensure we can validate the webhook admission bump
  • Kube 1.26 for publick8s:

[dduportal]

Operation on publick8s (wip)

Chart upgrades (falco and Nginx Ingress)

• [x] Fix Falco crashes on arm64 nodes by upgrading it to 3.x
  • Rationale: Falco needed a (major) upgrade as its daemonset images are crashing on the latest arm64 nodes of AKS 1.25.
  • PR: https://github.com/jenkins-infra/kubernetes-management/pull/4619
• [x] Upgrade Nginx ingress to latest patch version with revamped NetworkSecurities
  • Rationale: impact on updated/added ingresses can be detected long after the upgrade. Doing this before a full kube upgrade allows detecting issues immediately
  • PR: https://github.com/jenkins-infra/kubernetes-management/pull/4622

Public IP

• [x] Move Public IPs from MC_publick8s... to prod-public-ips resourcegroup (and delete manually their locks)
  • [x] public-publick8s-ipv4
  • [x] public-publick8s-ipv6
  • [x] ldap-jenkins-io-ipv4
• [x] Update Terraform Azure resources to (move public IP object + create new lock + add role assignment for AKS management)
  • [x] Disable Terraform Azure Job
  • [x] Apply manually (partial apply):
  • [x] public-publick8s-ipv4
  • [x] public-publick8s-ipv6
  • [x] ldap-jenkins-io-ipv4
  • [x] Enable Terraform Azure Job
  • [x] PR + (Sanity) checks should show no changes (or no major changes)
  • [x] Merge and deploy PR (main build green)
• [x] Update Kubernetes SVC LB configurations
  • [x] PR (and release) on LDAP jenkins-infra/helm-chart to support new Azure annotations
  • [x] PR (and release) on IPv6-LB jenkins-infra/helm-chart to support new Azure annotations
  • [x] Disable jenkins-infra/kubernetes-management job
  • [x] PR on jenkins-infra/kubernetes-management to update annotations and configuration for the 3 (used) SVC LB using AKS LB and (moved above) public IPs - https://github.com/jenkins-infra/kubernetes-management/pull/4647/files
  • [x] Enable jenkins-infra/kubernetes-management job
  • [x] Merge PR and deploy
  • [x] Sanity Checks

Kubernetes 1.26 upgrade

• Announcement: Thursday 09 Nov. 2023 at 08:30 UTC
  • [x] Open datetime announcement on status.jenkins.io - https://github.com/jenkins-infra/status/pull/415
  • [x] mailing lists - https://groups.google.com/g/jenkinsci-dev/c/Sf91vd1cOSA
  • [x] IRC #jenkins-infra - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$GfA9DgUrms_5A2npV2D4MEJfnQcTAI-AefZ_7x03wyg?via=libera.chat&via=gitter.im&via=matrix.org
• [x] Disable kubernetes management for publick8s cluster (remove it from Jenkinsfile_k8s) - https://github.com/jenkins-infra/kubernetes-management/pull/4652
• [x] Terraform PR to upgrade cluster to 1.26 - https://github.com/jenkins-infra/azure/pull/510
• [x] Merge and deploy Terraform PR
• [x] Wait for upgrade to be finished
• [x] Check terraform job and eventually trigger it for success with no changes
• [x] Sanity check with a local helmfile
• [x] Enable kubernetes management back for publick8s - https://github.com/jenkins-infra/kubernetes-management/pull/4653
• [x] Close announcements

Post Mortem

• 💡 When moving Azure resources from a resource group to another, do it sequentially. Yes it is slow but it avoids HTTP/500 error from the API followed by 10-15 min waiting anxiously before being able to retry ("eventually consistent" :trollface: )

[dduportal]

Mandatory logo

@smerle33 I'll let you close this issue with a mandatory gif or image ;)

[smerle33]