Upgrade to Kubernetes 1.25

[dduportal]

Previous upgrade: https://github.com/jenkins-infra/helpdesk/issues/3387

[github-actions[bot]]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% #3053
2. 92% #2930
3. 92% #2866
4. 77% #2664

[dduportal]

Updating kubectl:

• [x] Update the updatecli version tracking - https://github.com/jenkins-infra/docker-helmfile/pull/300
• [x] Updatecli automated update to the latest 1.25.x - https://github.com/jenkins-infra/docker-helmfile/pull/312
• [x] Deploy to jenkins-infra/kubernetes-management - https://github.com/jenkins-infra/kubernetes-management/pull/3962

[dduportal]

Since we have disabled doks due to Digital Ocean outage the 21 June 2023, we are taking the opportunity to upgrade both Digital Ocean clusters to 1.25 before putting back DigitalOcean clusters back to use.

Task list for both DigitalOcean clusters

• [x] No need to announce publicly as we already have an open incident

• [x] Changelog DO-specific: https://docs.digitalocean.com/products/kubernetes/details/changelog/. Other than the components upgrades (expected), 2 majors points:

  • Kubernetes (vanilla) general API removal for 1.25 (of deprecated APIs since 1.19/1.12): https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#urgent-upgrade-notes
  • DigitalOcean Control plane migration: https://docs.digitalocean.com/products/kubernetes/how-to/upgrade-cluster/#new-control-plane

• [x] Preparation based on the changelog:

  • No blockers
  • The detected deprecated APIs (using pluto) are only on DO-internal components (cilium-operatore and kube-dns). Recycling the worker nodes upgrades to new API: no action needed as the cluster upgrade will take care of it
  • There will be control-plane interruption so we're disabling management of API objects (no more helmfile until migration is complete => https://github.com/jenkins-infra/kubernetes-management/pull/4099
  • Since there will be control-plane interruption, given DigitalOcean gaves us credits only for testing the HA control plane, it looks a good incentive to enable it as part of the migration.

• [x] Upgrade both clusters to use HA control planes - https://github.com/jenkins-infra/digitalocean/pull/117

• [x] Upgrade doks using Terraform - https://github.com/jenkins-infra/digitalocean/pull/121

• [x] Upgrade doks-public using Terraform - https://github.com/jenkins-infra/digitalocean/pull/119

• [x] enable kubernetes-management and check it's working - https://github.com/jenkins-infra/kubernetes-management/pull/4099

• [x] Add doks back to ci.jenkins.io and test a pod with ACP - https://github.com/jenkins-infra/jenkins-infra/pull/2918

• [x] Close status and report here - https://github.com/jenkins-infra/status/pull/330

[dduportal]

Next step: upgrade of the AWS EKS clusters (including upgrade of components)

• [x] Plan a date for operation: 4th of July 2023 starting at 13:00 UTC
• [x] Check EKS changelog and verify any major changes (https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.25)
  • PSP removal from the API ✅ (we don't use it)
  • AWS ConfigMap enhancements 👁️ (need to be backuped and checked prior to the migration)
  • EndpointSlice APi changes requires https://github.com/jenkins-infra/kubernetes-management/pull/3838 to be done
  • Note strictly required, but the autoscaler will be upgraded prior to the migration (ref. https://github.com/jenkins-infra/kubernetes-management/pull/3963)
• [x] Announce
  • [x] to developers (no BOM builds during the migration) on mailing list - https://groups.google.com/g/jenkinsci-dev/c/WS8r1ZVdlec
  • [x] in status.jenkins.io - https://github.com/jenkins-infra/status/pull/340
• [x] Disable cik8s in ci.jenkins.io - https://github.com/jenkins-infra/jenkins-infra/pull/2949
• [x] Check and merge kubernetes-management PRs related to EKS components:
  • [x] "aws-load-balancer-controller" - https://github.com/jenkins-infra/kubernetes-management/pull/3838
  • [x] "autoscaler" - https://github.com/jenkins-infra/kubernetes-management/pull/3963
• [x] remove cik8s and eks-public from kubernetes-management - https://github.com/jenkins-infra/kubernetes-management/pull/4135
• [x] Migrate cik8s to 1.25.x - https://github.com/jenkins-infra/aws/pull/417
  • [x] Terraform for control plane + node pools
  • [x] Bump integrated modules
  • [x] Check manual access
• [x] Migrate eks-public to 1.25.x - https://github.com/jenkins-infra/aws/pull/417
  • [x] Terraform for control plane + node pools
  • [x] Bump integrated modules
  • [x] Check manual access
• [x] Add back cik8s and eks-public in kubernetes-management - https://github.com/jenkins-infra/kubernetes-management/pull/4136
  • ⚠️ Had the https://github.com/helm/helm/issues/11287 issue (helm tries to remove PSP while not existing in the API anymore 🤦 ). Solved by removing the eks namespace in both clusters, and let the jenkins-infra/kubernetes-management job recreate the releases.
  • We have to be careful to NOT have the same issue in AKS because deleting a release is not always doable...
  • 2 consecutive runs of the jenkins-infra/kubernetes-management job ✅
• [x] Checks:
  • [x] repo.aws.jenkins.io service is up and running
  • [x] No visible error logs in the autosclaer logs of cik8s
  • [x] No visible error logs in the autosclaer logs of eks-public
  • [x] No visible error logs in the LB controller logs of eks-public
• [x] Add back cik8s in ci.jenkins.io - https://github.com/jenkins-infra/jenkins-infra/pull/2951
  • [x] Include verifiying a full build with ACP - https://ci.jenkins.io/job/jenkinsci-libraries/job/jelly/job/PR-72/5/consoleFull
• [x] Close operation
  • [x] to developers on mailing list - https://groups.google.com/g/jenkinsci-dev/c/WS8r1ZVdlec/m/NFx47pVIBgAJ
  • [x] in status.jenkins.io

[dduportal]

AKS upgrade:

• [x] Plan a date for operation: 7th of July 2023 starting at 13:00 UTC

• [x] Check AKS changelog https://github.com/Azure/AKS/blob/master/CHANGELOG.md (searched for the 1.25 string), the following notable changes:

  • AKS begins pod security policy deprecation on 2022-11-01 API. The pod security policy will be removed completely on 2023-06-01 API with AKS 1.25 version or higher. You can migrate pod security policy to pod security admission controller before the deprecation deadline.

  • ✅ OK for us (click for details)

    ``` ➜ kubernetes-management git:(feat/disable-aks) kubectl config use-context privatek8s Switched to context "privatek8s". ➜ kubernetes-management git:(feat/disable-aks) kubectl get podsecuritypolicies.policy -A Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ No resources found ➜ kubernetes-management git:(feat/disable-aks) kubectl config use-context publick8s Switched to context "publick8s". ➜ kubernetes-management git:(feat/disable-aks) kubectl get podsecuritypolicies.policy -A Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ No resources found ```

  • Starting with Kubernetes 1.25, the host VM operating system will be Ubuntu 22.04 for Intel and ARM64 architectures

  • => ✅ https://github.com/jenkins-infra/helpdesk/issues/2982

  • Windows Server 2022 will be the default Windows host. Important, old windows 2019 containers will not work on windows server 2022 hosts.

  • ✅ https://github.com/jenkins-infra/azure/blob/7be8bfbbb101c5bb591b258eb0418fb478d56767/privatek8s.tf#L132 fixed to Windows2019 LGTM

  • Updated Calico to v3.23.3 when Kubernetes version is greater than or equal to v1.25.0.

  • => ✅ https://registry.terraform.io/providers/hashicorp/azurerm/3.64.0/docs/resources/kubernetes_cluster#network_policy we do not specify any network policy,n so no Calico (we should...)

  • Starting Kubernetes v1.25 two in-tree driver persistent volumes won't be supported in AKS : kubernetes.io/azure-disk, kubernetes.io/azure-file.

  • ✅ OK for us (Click to see details)

    - publick8s: ``` # Check for "in-tree" Disk volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'kubernetes.io/azure-disk' 0 # Check for CSI Disk volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'azuredisk' 55 # Check for "in-tree" blobfile volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'kubernetes.io/azure-file' 0 # Check for CSI blobfile volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'azurefile' 54 ``` - privatek8s ``` # Check for "in-tree" Disk volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'kubernetes.io/azure-disk' 0 # Check for CSI Disk volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'azuredisk' 54 # Check for "in-tree" blobfile volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'kubernetes.io/azure-file' 0 # Check for CSI blobfile volumes in all pods ➜ kubernetes-management git:(feat/disable-aks) kubectl get pod -A -o yaml | grep -c 'azurefile' 54 ```
  • Java/JDK support for cgroups v2 is available in JDK 11 (patch 11.0.16 and later) or JDK 15 and above. AKS Kubernetes 1.25+ uses cgroups v2. Please migrate your workloads to the new JDK.
  • => ✅ We use up to date java images
• [x] Announce
  • [x] in status.jenkins.io - https://github.com/jenkins-infra/status/pull/343
  • [x] in IRC
• [x] Check that there are NO PSP at all in the 2 clusters
• [x] remove publick8s and privatek8s from kubernetes-management - https://github.com/jenkins-infra/kubernetes-management/pull/4149
• [x] Migrate privatek8s to 1.25.x - https://github.com/jenkins-infra/azure/pull/427
  • [x] Disable Terraform Azure job
  • [x] Merge the PR on terraform azure
  • [x] Run upgrade from a local machine (infra.ci.jenkins.io can't upgrade itself)
  • [x] Wait for nodepool upgrades (Windows 2019 and Ubuntu to 22.04)
    • ⚠️ Same as https://github.com/jenkins-infra/helpdesk/issues/3387, we had to upgrade the nodepool manually through the Azure Portal WebUI (browse to node pool overview page, click on the "Kubernetes Version" and select the proper version
    • https://github.com/jenkins-infra/azure/pull/429 should be the solution
  • [x] Check access and the successful retstart of all services
  • [x] Enable and check the terraform azure jobs works again on infra.ci
  • [x] (async) try to allocate agents on release.ci with a job replay (dummy linux and a dummy window)
• [x] Migrate publick8s to 1.25.x - https://github.com/jenkins-infra/azure/pull/428
  • [x] Update, check and merge the PR
  • [x] Wait for nodepool upgrades (Windows 2019 and Ubuntu to 22.04)
  • [x] Check access and the successful restart of all services
  • ⚠️ Issues for some pods to reach ldap.jenkins.io: we realized that the pods were having IPs in the wrong subnet.
  • Opened https://github.com/jenkins-infra/azure/pull/431 to fix this but it fails to apply
  • AKS cluster was unresponsive due to a network overlap between the kubenet and the real subnet (ref. https://learn.microsoft.com/en-us/azure/aks/configure-kubenet#overview-of-kubenet-networking-with-your-own-subnet).
  • Add to re-create the cluster as we cannot change the pods CIDR without re-creating, causing an outage (see post mortem below).
  • [x] Check the terraform azure jobs works again on infra.ci
• [x] Add back publick8s and eks-privatek8s in kubernetes-management - https://github.com/jenkins-infra/kubernetes-management/pull/4152
• [x] Checks:
  • [x] No more pager duty alerts
  • [x] authentication (log off and log in) on accounts.jenkins.io works
• [x] Close operation - https://github.com/jenkins-infra/status/pull/347

[Gaoithe]

There is an outage. http://get.jenkins.io/ 13:29:49 UTC Friday, 7 July 2023 .. Okay, I see you are aware of outage. Good luck, hope you can fix and recover it without extreme stress, thank you!

[dduportal]

All the public services should be back. We are working on finishing the 1.25 post upgrade steps and we'll publish a post-mortem next week.

[dduportal]

Sub-tasks left beforer closing this issue:

• [x] Postmortem: write it down - https://github.com/jenkins-infra/jenkins.io/pull/6532
• [x] Postmortem: Add a lock to the public IP to avoid deletion when the clusters are deleted (which transitively removes the "automatic" resource group where the public_ip must be) - https://github.com/jenkins-infra/azure/pull/433
• [x] Postmortem: Put back javadoc to arm64 nodepool - https://github.com/jenkins-infra/kubernetes-management/pull/4160
• [x] Postmortem: put "as code" the setup of the infraciadmin service account used to administrate clusters (instead of a hidden script on my laptop...) - Tracked in https://github.com/jenkins-infra/helpdesk/issues/3679
• [x] Prepare the Kubernetes 1.26 issue with all the changes (and non changes) from this release - https://github.com/jenkins-infra/helpdesk/issues/3683

[timja]

which transitively removes the "automatic" resource group where the public_ip must be

not true, set this label: service.beta.kubernetes.io/azure-load-balancer-resource-group: myNetworkResourceGroup

https://learn.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address

[lemeurherve]

which transitively removes the "automatic" resource group where the public_ip must be

not true, set this label: service.beta.kubernetes.io/azure-load-balancer-resource-group: myNetworkResourceGroup

https://learn.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address

Thanks! We'll create a test IP in a resource group to check if we can safely move IPs to another resource group without recreating them, then we'll move prod IPs in a dedicated resource group (instead of the cluster node resource group) and add the label to the concerned services.

[dduportal]

Last step: https://github.com/jenkins-infra/helpdesk/issues/3683

[dduportal]

😱 Forgot the 1.25 logo:

Ref. https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/