Kubernetes upgrade from ver 1.27 to 1.28

[zhoujing2022]

Description Currently kyma supports k8s 1.27. As all dependencies are met we can support the next version 1.28.

SRE k8s upgrade plan: https://wiki.one.int.sap/wiki/pages/viewpage.action?pageId=3011925416

Furthermore the support for next version should be prepared by already having a post-main-k3d-k8s-preview pipeline in place to identify problems fast and start addressing them.

Tasks

• [x] [k8s update] @TW Pre-Notification to Customers via What's New. This Step must be done 4 weeks before Golive on PRD. (Before 14.Mar.2024)
• [x] ask every team about the impact from "Kubernetes Removals, Deprecations, and Major Changes in 1.28" - Ticket:#5057 and link every relevant changes in this ticket.
• [x] [k8s update] @zhoujing2022 Ensure no deprecated resources exist which will be no longer support on the new K8s cluster
• [x] [k8s update] @kyma-project/jellyfish Default version for Kyma CLI is k8s 1.28 (https://github.com/kyma-project/cli/issues/1936)
• [x] [k8s update] @neighbors Prow jobs are based on k8s 1.28 (see neighbours issue)
• [x] [k8s update] @neighbors Already existing release jobs (from last release) should be still using old k8s version. (see neighbours issue)
• [x] [k8s update] @neighbors Default version for kubectl is 1.28 (Docu, Prow) (see neighbours issue)
• [x] [k8s update] @TW document new version and add release notes ()

AC:

• all pipelines are upgraded to k8s 1.28
• CLI provisioning is adjusted to new versions
• documentation is updated
• migration guide with potential manual actions is ready
• there is a new k3d based pipeline running on k8s 1.28 for preview purposes

The list of pre-tasks: (POs analyze "Kubernetes Removals, Deprecations, and Major Changes in 1.28" - Ticket:???? and create/add the issues below. These Issues must be released before 13.Apr.2024 on PRD.) No Update from the teams.

Teams	Issues

Previous upgrade: https://github.com/kyma-project/kyma/issues/18352

Depends on https://github.com/kyma-project/test-infra/issues/6109

[janmedrek]

FYI: CLI should no longer be used for cluster provisioning, thus we can drop the part with the version bump there.

[zhoujing2022]

Released. Ticket is now closed.