Upgrade to Kubernetes 1.28

[dduportal]

Per https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar, Microsoft stops supporting Kubernetes 1.27 in July 2024.

As such, we must upgrade our Kubernetes clusters to 1.28.

Last issue for 1.27 was: https://github.com/jenkins-infra/helpdesk/issues/3948. Since last issue, we got rid of AWS and DigitalOcean clusters: we only have 4 Azure AKS clusters now.

---

Task list:

• [x] Upgrade kubectl within packer-images All-in-One
  • [x] First upgrade the updatecli manifest (https://github.com/jenkins-infra/packer-images/pull/1239)
  • [x] Then apply the updatecli PR that is produced by this change: (https://github.com/jenkins-infra/packer-images/pull/1240)
  • [x] Trigger a release of the packer-images (https://github.com/jenkins-infra/packer-images/releases/tag/1.73.1)
  • [x] Then apply the Kubernetes-Management PR that will use this new version -(https://github.com/jenkins-infra/kubernetes-management/pull/5331)

  • [x] Also apply the puppet (jenkins-infra/jenkins-infra) PR with the new version - (https://github.com/jenkins-infra/jenkins-infra/pull/3485)

• [x] Check AKS changelog for 1.28 - https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2023-11-05
• [x] Upgrade AKS cluster infracijio-agents-1 (details)
• [x] Upgrade AKS cluster cijenkinsio-agents-1 (details)
• [x] Upgrade AKS cluster privatek8s (details)
• [ ] Upgrade AKS cluster publick8s (details)

[github-actions[bot]]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% #3683
2. 92% #3582
3. 92% #3387
4. 92% #3053
5. 92% #2930

[smerle33]

infracijio-agents-1

• [x] Announcement: no announcement needed as the cluster is not used yet
• [x] Disable kubernetes management for infracijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - https://github.com/jenkins-infra/kubernetes-management/pull/5337
• [x] Terraform PR to upgrade cluster to 1.28 but NO merge - https://github.com/jenkins-infra/azure/pull/738
  • [x] checks that all checks are ok
  • [x] Merge the PR
• [x] Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - https://github.com/jenkins-infra/kubernetes-management/pull/5338
• [x] Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1247/

🎉

[smerle33]

cijioagents1

We plan to handle this upgrade Tuesday 25 of june morning Paris time

• [x] Announcements:
  • [x] status.jenkins.io - https://github.com/jenkins-infra/status/pull/508 - @dduportal and @smerle33
  • [x] developer mailing list + jenkins-infra mailing - @dduportal and @smerle33
  • [x] IRC #jenkins-infra - @dduportal and @smerle33
  • [x] Add a banner on ci.jenkins.io - @dduportal and @smerle33
• [x] Disable kubernetes management for cijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - https://github.com/jenkins-infra/kubernetes-management/pull/5348 - @jayfranco999
• [x] Disable the 2 clouds defined in ci.jenkins.io on puppet (jenkins-infra/jenkins-infra) for the operation ci.jenkins.io-agents-1-bom and ci.jenkins.io-agents-1 - https://github.com/jenkins-infra/jenkins-infra/pull/3495 - @jayfranco999
• [x] Terraform PR to upgrade cluster to 1.28.9 but NO merge - https://github.com/jenkins-infra/azure/pull/750 - @jayfranco999
  • [x] checks that all checks are ok
  • [x] Merge the PR
• [x] Re-Enable the 2 clouds puppet defined - https://github.com/jenkins-infra/jenkins-infra/pull/3496 - @jayfranco999
• [x] Try the connexion to the Clouds for this cluster within the ci.jenkins.io UI
• [x] Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - - @jayfranco999 and @smerle33
• [x] Check terraform job and eventually trigger it for success with no changes - - @jayfranco999 and @smerle33
• [x] Close Annoucements - @jayfranco999 and @smerle33 and @dduportal
  • [x] status.jenkins.io - https://github.com/jenkins-infra/status/pull/509
  • [x] developer mailing list + jenkins-infra mailing - @dduportal and @smerle33
  • [x] IRC #jenkins-infra - @dduportal and @smerle33
  • [x] Remove the banner on ci.jenkins.io - @dduportal and @smerle33

---

POSTMORTEM

we need to ensure that puppet catch up the change to disable and to re-enable the cloud cluster. The disable is double handled by the banner in ci.jenkins.io but still ...

[smerle33]

AZURE

Update: AKS Upgrade plan privatek8s

privatek8s

• Announcement: date/time UTC
  • [x] Open datetime announcement on status.jenkins.io - https://github.com/jenkins-infra/status/pull/512
  • [x] Jenkins-infra mailing lists - https://groups.google.com/g/jenkins-infra/c/0s44tR8EnXQ
  • [x] IRC #jenkins-infra
• [x] Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - https://github.com/jenkins-infra/kubernetes-management/pull/5353
• [x] Terraform PR to upgrade cluster to 1.28 - https://github.com/jenkins-infra/azure/pull/756
• [x] Merge Terraform PR and let the upgrade happen
• [x] Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1270/
• [x] Enable kubernetes management for privatek8s cluster and check it runs successfully - https://github.com/jenkins-infra/kubernetes-management/pull/5362
• [x] Check for services healthiness (pagerduty, datadog, manual control, logging in on infra.ci.jenkins.io/release.ci.jenkins.io with the Infra Agents Health job https://release.ci.jenkins.io/job/Infra%20Agents%20Health/)
• [x] Close announcements

[smerle33]

Update on publick8s Azure AKS upgrade: plan Thursday 4 March at 07h00 UTC

• [x] Changelog for Kube 1.28 to be checked with a focus on kubernetes/kubernetes@master/CHANGELOG/CHANGELOG-1.28.md#urgent-upgrade-notes
• [x] Changelog focus on Azure : https://github.com/Azure/AKS/blob/master/CHANGELOG.md?rgh-link-date=2024-06-18T11%3A54%3A32Z#release-2023-11-05
• Announcement: Thursday 4 March at 07h00 UTC
  • [x] status.jenkins.io - https://github.com/jenkins-infra/status/pull/520 - @jayfranco999
  • [ ] developer mailing list (Authentication outage due to LDAP restart) + jenkins-infra mailing - - @dduportal
  • [ ] #jenkins-infra Matrix IM channel - - @jayfranco999
• [ ] Disable kubernetes-management for publick8s cluster - - @jayfranco999
• [ ] Bump Kube version to 1.28 for publick8s - - @jayfranco999
• [ ] Terraform jenkins-infra/azure project has a successful build without changes on the main branch after the upgrades - - @jayfranco999 @dduportal @smerle33
• [ ] Check the LDAP restart successfully once upgrade is performed. If the pod restarted and is running, try the service by logging-in on ci.jenkins.io and repo.jenkins-ci.org. @jayfranco999 @dduportal @smerle33
• [ ] Check the Artifact Caching Proxy ("ACP") service in azure/publick8s is up and running use this link https://repo.azure.jenkins.io/public/org/jenkins-ci/plugins/plugin/4.61/ @jayfranco999 @dduportal @smerle33
• [ ] Enable kubernetes-management back for publick8s - - @jayfranco999
• [ ] Close announcements
  • [ ] status.jenkins.io - - @jayfranco999
  • [ ] developer mailing list (Authentication outage due to LDAP restart) + jenkins-infra mailing - - @jayfranco999
  • [ ]#jenkins-infra Matrix IM channel - - @jayfranco999