Kubernetes upgrade from ver 1.26 to 1.27

[zhoujing2022]

Description Currently kyma supports k8s 1.26. As all dependencies are met we can support the next version 1.27.

SRE k8s upgrade plan: https://wiki.one.int.sap/wiki/pages/viewpage.action?pageId=3011925416

Furthermore the support for next version should be prepared by already having a post-main-k3d-k8s-preview pipeline in place to identify problems fast and start addressing them.

Tasks

• [x] [k8s update] @TW Pre-Notification to Customers via What's New. This Step must be done 4 weeks before Golive on PRD. (Upgrade-Date on PRD: 30.Oct.- 02.Nov.2023 )
• [x] ask every team about the impact from "Kubernetes Removals, Deprecations, and Major Changes in 1.27" - Ticket:4140 and link every relevant changes in this ticket.
• [x] [k8s update] @zhoujing2022 Ensure no deprecated resources exist which will be no longer support on the new K8s cluster - Ticket:4140
• [x] [k8s update] @kyma-project/jellyfish Default version for Kyma CLI is k8s 1.27 (https://github.com/kyma-project/cli/issues/1855)
• [x] [k8s update] @neighbors Prow jobs are based on k8s 1.27 (see neighbours issue)
• [x] [k8s update] @neighbors Already existing release jobs (from last release) should be still using old k8s version. (see neighbours issue)
• [x] [k8s update] @neighbors Default version for kubectl is 1.27 (Docu, Prow) (see neighbours issue)
• [x] [k8s update] @TW document new version and add release notes (https://github.com/kyma-project/kyma/pull/)
• [x] @huskies update kube-state-metrics (not needed any more. Monitoring will be removed with rel.2.20 - before the K8s upgrade)

AC:

• all pipelines are upgraded to k8s 1.27
• CLI provisioning is adjusted to new versions
• documentation is updated
• migration guide with potential manual actions is ready
• there is a new k3d based pipeline running on k8s 1.27 for preview purposes

The list of pre-tasks: (POs analyze "Kubernetes Removals, Deprecations, and Major Changes in 1.27" - Ticket:4140 and create/add the issues below. These Issues must be released before 30.10.2023 on PRD.) No Update from the teams.	Teams	Issues
4218	framefrog

Previous upgrade: https://github.com/kyma-project/kyma/issues/17018

Depends on https://github.com/kyma-project/test-infra/issues/6109

[zhoujing2022]

As discussed with Neighbors. 2 tasks are removed:

[k8s update] [@neighbors](https://github.com/neighbors) kyma-upgrade-gardener-kyma2-minor-versions job should be still using old k8s version. (see [neighbours issue](https://github.com/kyma-project/test-infra/issues/9185))

• this can be deleted

[k8s update] [@neighbors](https://github.com/neighbors) Setup preview job that will verify if kyma can run with k8s1.27 (next k8s version), create tickets (and link them) for other teams if the pipeline gets red (see [neighbours issue](https://github.com/kyma-project/test-infra/issues/9185))

• this can be also deleted, there is no preview job in test-infra nymore

[zhoujing2022]

It went live on 27./28.Jan.2024. Issue will be closed.