[ci:component:github.com/gardener/gardener:v1.69.3->v1.82.1]

Release Notes:

The `MachineClassKind()`, `MachineClass()`, and `MachineClassList()` methods have been dropped from the generic `Worker` actuator's interface and do not need to be implemented anymore.

The `shoots/adminkubeconfig` relies on the `ca-client` `InternalSecret` only and does not use the `ShootState` object anymore.

`kubectl proxy` now works as expected in the local development setup in conjunction with highly available vpn

The following image is updated:
- quay.io/prometheus/prometheus: v2.41.0 -> v2.43.1

The gardenlet and the gardener-operator will now use the new `service.kubernetes.io/topology-mode=auto` annotation when enabling topology-aware routing for a Service when the Kubernetes version of the runtime cluster is >= 1.27. In Kubernetes 1.27, the `service.kubernetes.io/topology-aware-hints=auto` annotation is deprecated in favor of the newly introduced `service.kubernetes.io/topology-mode=auto`

File ownership for `var/etcd/data` will be changed to non-root user (65532).

Functions `controllerutils.GetAndCreateOrMergePatch`, `controllerutils.GetAndCreateOrStrategicMergePatch`, `controllerutils.CreateOrGetAndMergePatch` and `controllerutils.CreateOrGetAndStrategicMergePatch` were incompatibly changed and now accept a `controllerutils.PatchOption` instead of `client.MergeFromOption`.
If your controllers use one of these functions with `client.MergeFromOption`, you should update it to `controllerutils.PatchOption`.
The `controllerutils.PatchOption` can hold two options today:
- `client.MergeFromOption` which is passed to the underlying patch function.
- `controllerutils.SkipEmptyPatch` which prevents sending empty patches (`{}`).

`nginx-ingress-controller` image is updated to `v1.9.3`.

`gardener-operator` maintains the two most recent `generic-token-kubeconfig` secrets in the runtime-cluster. In addition the latest secret name is published to the `garden` resource in `.metadata.annotations[generic-token-kubeconfig.secret.gardener.cloud/name]`. Third-party components referring to this secret should check this annotation value after a credentials or CA rotation for the virtual-garden cluster took place.

Extensions vendoring this `gardener/gardener` version need to provide RBAC privileges for `PATCH apps/depoyments/scale`.

Improves testing flakiness of logging testmachinery test by making the loki init-container reliable.

The Plutono version has been updated from `v7.5.23` to `v7.5.24`.

It is now possible to provide namespace selectors for additional namespaces which should be covered by the `NetworkPolicy` controllers of `gardener-operator` or `gardenlet`. The selectors must be provided via their component configs. Please consult [this document](https://github.com/gardener/gardener/blob/master/docs/usage/network_policies.md#additional-namespace-coverage-in-gardenseed-cluster) for further insights.

Added new option to `./hack/generate-controller-registration.sh` script `[-e, --pod-security-enforce[=pod-security-standard]` which sets the `security.gardener.cloud/pod-security-enforce` annotation of the generated `ControllerRegistration`. When not set this option defaults to `baseline`.

`gardenlet` is taking over management of the `CustomResourceDefinition`s for the `machine.sapcloud.io/v1alpha1` API group, hence extensions do no longer need to take care. Consequently, the `extensions/pkg/controller/worker.Options` struct as well as the `extensions/pkg/controller/worker.ApplyMachineResources{ForConfig}` functions are deprecated and will be removed in a future release.

There is now a new script (`hack/check-skaffold-deps-for-binary.sh`) that can be used by gardener extensions to validate their skaffold ko dependencies.

Add a learner with backoff in case of scale-up feature is triggered.

`operator` now deletes `ManagedResources` deployed to the virtual-garden before deleting `virtual-garden-kube-apiserver`.

Before upgrading to this gardener version, operators should configure `gardener-apiserver` to encrypt the `internalsecrets.core.gardener.cloud` resource in etcd.

An issue has been fixed for highly-available `Shoot`s whose `etcd` clusters didn't get ready in the `Completing` phase of a CA credentials rotation.

The deprecated local development setups have been removed. From now on, only the `kind`-based setups are supported. Please refer to [this guide](https://github.com/gardener/gardener/blob/master/docs/development/local_setup.md) for all information.

Prevent nil pointer exceptions on shoot deletion in `gardenlet` when seed namespace is gone.

Suppress the event-logger `nest` filter's warnings in the fluent-bit.

Add failure tolerance option to the `CreateShoot` test.

⚠️ Gardener does no longer support garden, seed, or shoot clusters with Kubernetes versions < 1.24. Make sure to upgrade all existing clusters before upgrading to this Gardener version.

`gardener-operator` is now managing the Gardener control plane components (`gardener-{apiserver,admission-controller,controller-manager,scheduler}`).

The `Shoot` maintenance controller now updates the CRI of worker pools from `docker` to `containerd` when force-upgrading from Kubernetes `v1.22` to `v1.23`.

A bug causing the managedseed controller to error if the controller restarts and the seed secret is already deleted is now fixed.

`gardenlet` will no longer respect `ConfigMap`s labeled with `extensions.gardener.cloud/configuration=logging`. The way to deploy a new filter or parser configuration is to create `ClusterFilter`s or `ClusterParser`s custom resources in the seed cluster.

A bug in the local development environment has been fixed which prevented admission of Gardener resources by extension webhooks.

The `Secrets` type as well as the `Delete` functions for secrets were removed from `pkg/utils/managedresources/builder` since their usage was prone to errors. The higher level package `pkg/utils/managedresources` should be used instead.

A bug has been fixed which was allowing users to specify an extension of the same type in `.spec.extensions[].type` more than once in the `Shoot` API.

The `gardener-apiserver` now drops expired `Kubernetes` and `MachineImage` versions from `Cloudprofile`s during creation.

Missing permissions were added for the Gardenlet service account for `Machine` objects. This fix is relevant if feature gate `MachineControllerManagerDeployment` is enabled in your landscape.

The `core/v1alpha1` API version is dropped. Make sure that you don't use the `core/v1alpha1` API version in your machinery.

Update to golang v1.21

Change port of ssh reverse tunnel to 443

Added check to ensure that the scale up annotation is removed from the etcd statefulset only when scale-up succeeds

A bug that prevented finalizers from being added to referenced `Secret`s or `ConfigMap`s in `.spec.resources` in `Shoot`s has been fixed.

A guideline for developers regarding [`TODO` statements](https://github.com/gardener/gardener/blob/docs/master/development/process.md#todo-statements) has been introduced.

A new field `errorCodeCheckFunc` is introduced in the generic `Worker` actuator. This should be set to parse the Gardener error codes from the error returned in `Worker` reconciliation.

A bug has been fixed which caused `ServiceAccount`s related to garden access secrets for extensions to leak in the seed namespace in the garden cluster after uninstallation of said extensions.

It is now possible to annotate managed resources part of `ManagedResource` objects with `resources.gardener.cloud/finalize-deletion-after=<duration>`, e.g., `resources.gardener.cloud/finalize-deletion-after=1h`. After this time, `gardener-resource-manager` will forcefully delete the resource by removing their finalizers.

The `terraformer` library will now skip deletion of the Terraformer pod when the request context has been canceled. This change aims to prevent inconsistencies in Terraform state by attempting to allow uninterrupted execution of healthy Terraformer pods.

The deprecated `allow-{to,from}-shoot-apiserver` `NetworkPolicy`s have been dropped. Ensure that all registered extensions have been adapted.

`pkg/utils/chart` does now support embedded charts. The already deprecated methods in the `ChartApplier` and `ChartRenderer` will be removed in a few releases, so extensions should adapt to embedded charts.

The `NetworkPolicy` reconciler is only added to `gardener-operator` if the `.spec.runtimeCluster.networking.{pods,services}` fields of the `Garden` are set.

Remove unneeded Monitor function from iptables implementation

Print build version and go runtime info.

Upgraded Ginkgo v1 to v2 and updated other dependencies

Update Prometheus job `tunnel-probe-apiserver-proxy` to fix for HA VPN mode

The obsolete `addons` `ManagedResource` is now properly cleaned up.

Machine scale-up delay for new pods can now be configured for `cluster-autoscaler` via the field `.spec.kubernetes.clusterAutoscaler.newPodScaleupDelay` in the `Shoot` API .

Bump builder image golang from `1.20.2` to `1.20.4`

Update local-setup to `kind@v0.18.0`.

Remove lastOperation check in fluent-bit-to-vali plugin.

Shoot addon `nginx-ingress-controller` image is updated to `v1.8.0` for Kubernetes `v1.24+` clusters, to `v1.6.4` for Kubernetes `v1.23` clusters, and to `v1.4.0` for Kubernetes `v1.22` clusters.

Webhooks remediator now sets the timeoutSeonds to 3 seconds for webhook affecting lease resources in `kube-system` namespace.

Now git revision and commit ids are properly propagated through build variables. These are showed in the fluent-bit plugin logs during start.

Fixes for `make check` target

Fixed an issue that would cause the `gardenlet` to run into `CrashLoopBackoff` when following the docs/development/getting_started_locally.md#remote-local-setup guide.

Fixed flaky operator behaviour with regards to istio deployment caused by concurrent update of garden object

Base alpine image for etcd-custom-image upgraded from `3.15.7` to `3.15.8`

It is now possible to enable disabled APIs for workerless shoot clusters via `spec.kubernetes.kubeAPIServer.runtimeConfig`.

A bug causing the shoot provider label in the infrastructure secret to not get cleaned up is now fixed.

A new field `.spec.virtualCluster.dns.domains` was added to the `Garden` API. This field allows to expose the `kube-apiserver` of the virtual cluster via multiple domains. Earlier, the API only accepted one domain name via `.spec.virtualCluster.dns.domain`.
⚠️ With this change `.spec.virtualCluster.dns.domain` is deprecated and will be removed in the next release. Please update your `Garden` resource to the new `.spec.virtualCluster.dns.domains` field by removing the existing domain configuration from `dns.domain` and add it as the first entry of `dns.domains`.

The `alpha.kube-apiserver.scaling.shoot.gardener.cloud/class` annotation on `Shoot`s has no effect anymore and should be removed.

The `gardenlet`'s `ManagedSeed` controller now cleans up the referred seed secret when `.spec.secretRef` is unset in the seed template.

Upgrade to go 1.20.3

Included `UnavailableReplicas` in determining if a machine deployment status update is needed

A bug has been fixed that prevented `ControllerInstallation`s from getting deleted when the backing `ControllerRegistration` with `.spec.deployment.policy={Always,AlwaysExceptNoShoots}` was deleted.

Update golang image in verify step to 1.21.3.

A new feature gate named `ContainerdRegistryHostsDir` is introduced to gardenlet. When enabled, the `/etc/containerd/certs.d` directory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the [registry-cache extension](https://github.com/gardener/gardener-extension-registry-cache/) will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation).

Upgrade gardener/gardener from `1.65.0` to `1.76.0`

Etcd-related secrets will now be mounted onto the `/var/` directory instead of `/root/`.

Fixed a bug that caused HVPA reconciliation to fail with `expected pointer, but got v2beta1.MetricSpec type` when the HPA spec had changed.

Multiple expanders for `cluster-autoscaler` can now be specified in the `Shoot` API via the `.spec.kubernetes.clusterAutoscaler.expander` field.

The `node-local-dns` `ConfigMap` now has a label `k8s-app=node-local-dns` for identifying it.

`gardenlet` and `gardener-operator` managed `deployment`s and `statefulset`s can now be equipped with toleration seconds for taints `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`.
Please consult the respective component config examples ([`gardenlet`](https://github.com/gardener/gardener/blob/master/example/20-componentconfig-gardenlet.yaml), [`gardener-operator`](https://github.com/gardener/gardener/blob/master/example/operator/10-componentconfig.yaml)) for more information.

Gardener now supports seed clusters with Kubernetes versions up to `v1.26`.

The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected.

An issue has been fixed that caused traffic from outside of the cluster to `Istio-Ingress` being blocked. This is only relevant if seed(s) specify additional load balancer annotations via `seed.spec.settings.loadBalancerServices.annotations`.

Default log level in fluent-bit is changed from `info` to `error`

The deprecated `ChartRenderer.Render` and `ChartApplier.{Apply,Delete}` methods have been dropped. Use `ChartRendere.RenderEmbeddedFS` and `ChartApplier.{Apply,Delete}FromEmbeddedFS` instead.

`gardener-operator` now refuses to start if operators attempt to downgrade or skip minor Gardener versions. Please see [this document](https://github.com/gardener/gardener/blob/master/docs/deployment/version_skew_policy.md) for more information.

Backup-restore waits for its etcd to be ready before attempting to update peerUrl

It is now easier to annotate `Service`s related to extensions serving webhook handlers that must be reached by `kube-apiserver`s running in separate namespaces such that the respective network traffic gets allowed. Please refer to [this guide](https://github.com/gardener/gardener/blob/master/docs/usage/network_policies.md#webhook-servers) for all information. Extensions serving shoot webhook should make use of this new approach - the old functionality deploying dedicated `NetworkPolicy`s is deprecated and will be removed in the future.

Eliminated `Role` helm charts and converted into Golang component with added unit tests.

Add Prometheus alert for pending seed pods

The testmachinery tests now use `AdminKubeconfig` of the `Shoot`s of `ManagedSeed`s to create seed client.

`gardener-operator` now renews garden access secrets and the gardenlet kubeconfig on all `Seed`s during CA/service account signing key credentials rotation.

Following dependency has been updated:- 
- github.com/gardener/etcd-druid v0.18.1 -> v0.18.4

Before upgrading to this Gardener versions, you must make sure that the `Service`s of all registered provider extensions serving webhooks for the shoot cluster are annotated with `networking.resources.gardener.cloud/from-all-webhook-targets-allowed-ports=[{"protocol":"TCP","port":<port>}]`, `networking.resources.gardener.cloud/namespace-selectors=[{"matchLabels":{"gardener.cloud/role":"shoot"}}]`, and `networking.resources.gardener.cloud/pod-label-selector-namespace-alias=extensions`.

Support for `nip.io` shoot domains is discontinued.

The `hack/generate-crds.sh` script now receives the file name prefix via the `-p` option (previously, the prefix was the first argument to the script).

The promoted or deprecated feature gates `ManagedIstio` and `ReversedVPN` have been removed. Remove these feature gates before updating to this version of Gardener.

A configuration issue that resulted in a relatively slow startup and termination of the vali pods is fixed.

The deprecated `identity` value is no longer passed when `ControllerInstallation` Helm charts are deployed.

It is now possible to reference `Secret`s containing kubeconfigs for admission plugins in `Shoot`s. The referenced `Secret` must be referenced in`.spec.resources` as well as in `.spec.kubernetes.kubeAPIServer.admissionPlugins[].kubeconfigSecretName`.

Gardener can now support clusters with Kubernetes version 1.28. Extension developers have to prepare individual extensions as well to work with 1.28.

Updated golang container image build version to 1.20.4

The `gardener-operator` does now also manage `kube-state-metrics`.

The following golang dependencies have been upgraded, please consult the upstream release notes and [this issue](https://github.com/gardener/gardener/issues/8382) for guidance on upgrading your golang dependencies when vendoring this gardener version:
- `k8s.io/*` to `v0.28.2`
- `sigs.k8s.io/controller-runtime` to `v0.16.2`
- `sigs.k8s.io/controller-tools` to `v0.13.0`

Bump `k8s.io/*` deps to v0.27.2

In order to allow `kube-apiserver` pods of shoot or garden clusters to reach webhook servers, they must no longer be explicitly labeled with `networking.resources.gardener.cloud/to-<service-name>-<protocol>-<port>=allowed`. Instead, it is enough to annotate the `Service` of the webhook server with `networking.resources.gardener.cloud/from-all-webhook-targets-allowed-ports=<ports>`.

The skaffold version is updated from v2.7.0 to v2.8.0.

github.com/gardener/etcd-druid #714 @aaronfern
Alpine image used in init containers is now part of the IMAGEVECTOR_OVERWRITE

The reconciliation time limit for the controller resource reconciliation, e.g. for `ManagedResource`, has been increased from `1m` to `3m`.

A bug causing `EtcdCopyBackupsTask` jobs to fail to create temp snapshot directory while using distroless etcd-backup-restore image `v0.25.x` has been resolved.

Updated cluster-proportional-autoscaler to v1.8.8

If `gardenlet` is responsible for a managed `Seed`, it will delete all `ShootState` resources for its `Shoot`s that are not currently in migration. See also [GEP-22](https://github.com/gardener/gardener/blob/master/docs/proposals/22-improved-usage-of-shootstate-api.md) for further details about the motivation.

The following images are updated:
- registry.k8s.io/metrics-server/metrics-server: v0.6.3 -> v0.6.4
- registry.k8s.io/cpa/cluster-proportional-autoscaler: v1.8.8 -> v1.8.9
- registry.k8s.io/coredns/coredns: v1.10.0 -> v1.10.1
- quay.io/prometheus/blackbox-exporter: v0.23.0 -> v0.24.0
- quay.io/prometheus/node-exporter: v1.5.0 -> v1.6.1
- ghcr.io/credativ/plutono: v7.5.22 -> v7.5.23
- ghcr.io/prometheus-operator/prometheus-config-reloader: v0.61.1 -> v0.67.1
- registry.k8s.io/dns/k8s-dns-node-cache: 1.22.20 -> 1.22.23

`custodian-sync-period` value is set to `15s` in the Helm chart for etcd-druid.

Annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns` is deprecated and will be removed in future releases. Use field `.spec.systemComponents.nodeLocalDNS.enabled` in `Shoot` instead. Switching on node-local-dns via shoot specification will roll the nodes even if node-local-dns was enabled beforehand via annotation.

Use cgroupv2 fix for local-setup on macOS too.

Bump alpine base image from `1.16.3` to `1.16.5`

The Seed's `.spec.settings.ownerChecks` field is now no-op - the `gardener-apiserver` no longer defaults this field and no longer validates it. The field will be set always to `nil` on CREATE/UPDATE request.
Gardener landscape operators specifying this field should no longer specify it. The field will be removed in a future version of Gardener.

Test-machinery integration tests are now using upstream K8s e2e test images such as `registry.k8s.io/e2e-test-images/busybox`, `registry.k8s.io/e2e-test-images/agnhost` instead Gardener images such as `eu.gcr.io/gardener-project/3rd/busybox`, `eu.gcr.io/gardener-project/3rd/alpine` and others.

Operators can now use the annotation `gardener.cloud/operation=rotate-observability-credentials` on the `garden` resource to rotate the observability credentials.

Gardener denies setting `Shoot.Spec.ControlPlane.HighAvailability.FailureTolerance.Type` if shoot is hibernated.

Gardener can now support clusters with Kubernetes version 1.28. In order to allow creation/update of 1.28 clusters you will have to update the version of your provider extension(s) to a version that supports 1.28 as well. Please consult the respective releases and notes in the provider extension's repository.

Stability of the ssh tunnel in the local extension setup should improve due to better failure handling.

The `github.com/golang/mock/gomock` dependency is replaced by `go.uber.org/mock`.

Introduced a new field called `machineDeploymentsLastUpdateTime` in the `Worker` status to keep track of the time when the status of the Worker resource was last updated with the latest machine deployments.

Go version is updated to 1.20.4.

It is now possible to trigger gardenlet kubeconfig renewal for unmanaged `Seed`s by annotating them with `gardener.cloud/operation=renew-kubeconfig`. This was already supported for `ManagedSeed`s only.

The `pkg/operation/botanist/component/*` resources have been moved to `pkg/component/*`.

Update Kubernetes dependencies (especially `k8s.io/client-go`) from `v0.26.3` to `v0.26.4` to resolve panic on working with special shoots.

The `alpha.featuregates.shoot.gardener.cloud/apiserver-sni-pod-injector` annotation has been dropped and is no longer available for `Shoot`s. It should be removed from all existing `Shoot` resources.

Block public access for S3 buckets created by e2e tests.

When performing control plane migration with `provider-local`, the full migration and restoration logic implemented in the extensions library (generic `Worker` actuator) is now executed (previously, it was skipped). This improves the accuracy of the e2e tests for control plane migration.

A bug where MCM removed a machine other than the one , CA wanted , is resolved.

A bug preventing `prometheus` ingress to use `wildcard-certificate` is fixed.

* More categories are added to label a release note for a PR on DWD.
* Release notifications would now be sent to `gardener-dwd` channel (private) on releases.

Gardener now deploys the [cluster-autoscaler](https://github.com/gardener/autoscaler) earlier in the shoot reconciliation flow without checking if the worker pools are ready.

On deletion, the generic `ControlPlane` actuator will now redeploy the cloud config chart to allow provider extensions update the content with the most up-to-date information.

The `DisableScalingClassesForShoots` feature gates has been promoted to GA (and is now always enabled).

gardener-apiserver now exposes a new `core.gardener.cloud/v1beta1.InternalSecret` API, see the [documentation](https://github.com/gardener/gardener/blob/master/docs/concepts/apiserver.md#internalsecrets) for more information.

`kubectl get garden` now features additional printer columns providing more information about the substantial configuration values and statuses.

Add CVE categorization for etcd-backup-restore.

The `FullNetworkPoliciesInRuntimeCluster` feature gate has been promoted to beta and is now turned on by default. Before deploying this Gardener version, make sure that all your registered extensions support this feature gate.

`gardener-operator` now takes over management of `gardener-metrics-exporter`.

The Istio Ingress-Gateway deployment was refined to speed up seed bootstrapping times.

Removed `service.beta.kubernetes.io/aws-load-balancer-type: nlb` annotation from istio-ingressgateway service template. Set this annotation in seed configuration. Note: Changing load balancer type creates a new one, old one requires manual clean-up.

Adapt vpa-updater QPS limits such that it doesn't get throttled on large clusters

A bug has been fixed for Istio-Ingress Gateways for seeds that use `ExposureClassHandler`s. Earlier, annotations in `seed.spec.settings.loadBalancerServices` caused an override of the ones specified in `gardenletConfiguration.exposureClassHandler[].loadBalancerService` for zonal Istios. Now, annotations in `gardenletConfiguration.exposureClassHandler[].loadBalancerService` are given priority, like it was already the case of the global Istio.

`nginx-ingress-controller` image is updated to `v1.9.0`.

Bump alpine base version for Docker build to `3.18.2`.

Update alpine base image version to 3.18.3.

Shoot node network and seed pod network need to be disjoint. This will be checked during scheduling of a shoot cluster, i.e. during initial admission or on control-plane migration.

An issue has been fixed which caused CoreDNS to not rewrite CNAME values in DNS answers.

It is possible to delete a Shoot even if `shoot.gardener.cloud/ignore` annotation is set to true.

Makefile has been updated to use `Skaffold` for deploying `etcd-druid` with the `make deploy` target, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing.

Use admission v1 instead of v1beta1 for apiserver-proxy webhook.

A bug which was causing race conditions to occur during reconciliation of extension resources was fixed.

Updated go to 1.20.7

⚠️ Seeds' `.spec.settings.ownerChecks.enabled` field is locked to `false` (i.e. if the field value is true a validation error will be returned). Before updating to this version of Gardener, set `.spec.settings.ownerChecks.enabled` field to `false` for you Seeds and ManagedSeeds.

Developer Action Required: The `make deploy` command has been replaced with `make deploy-via-kustomize`. Please update your deployment workflows accordingly.

An issue causing panic in the health check for extension, when the health check result is empty, is fixed.

All default images are now present in `images.yaml`

Update golang 1.20.4 -> 1.21.3

`nginx-ingress-controller` now enables annotation validation.

The `MachineControllerManagerDeployment` has been promoted to GA and is now locked to "enabled by default". Make sure that all registered provider extensions support this feature gate before upgrading to this version of Gardener.

Configure the value for the flag `metrics-scrape-wait-duration` for compaction controller to set a wait duration at the end of every compaction job, to allow for metrics to be scraped by a Prometheus instance.

Update alpine base image version to 3.18.4.

When scaling from single-node to multi-node etcd cluster, Etcd Druid will now first ensure that any change to the peer URL (e.g TLS enablement)  is seen by the existing etcd process running within the etcd member pod. Once that is confirmed then it will scale up the Etcd StatefulSet and add relevant annotations.

An issue has been fixed which was causing a broken `ControlPlaneHealthy` condition report for `Shoot`s when the `MachineControllerManagerDeployment` feature gate gets enabled until their next reconciliation.

gardener-apiserver: The kubelet version constraint validation is now fixed to also cover the Shoot K8s version update. Previously it was possible to update the Shoot K8s version to a new minor version when the Shoot has a worker pool with machine image version which kubeletVersionConstraint does not match the new K8s version.

`pkg/resourcemanager/controller/garbagecollector/references.InjectAnnotations` now also handles `pods.spec.imagePullSecrets`.

Upgraded `etcd-backup-restore` from `v0.24.3` to `v0.24.6` for `etcd-custom-image`, and from `v0.25.1` to `v0.26.0` for `etcd-wrapper`

Update `k8s.io/client-go` from v0.17.0 to v0.26.2

`Secret`s/`ConfigMap`s referenced in `.spec.resources` of `Shoot`s are now protected with a finalizer to ensure they do not disappear from the system as long as they are still referenced somewhere.

Fixed a possibility for the `migrate` phase of control plane migration to become permanently stuck if the shoot was created when the `MachineControllerManagerDeployment` feature gate is disabled, control plane migration is triggered for the shoot and the feature gate is enabled during the migration phase.

Shoot fields `.spec.dns.providers[].domains` and `.spec.dns.providers[].zones` are now deprecated and expected to be removed in version `v1.87`. Please use the extensions' configuration to configure providers with this ability.

Initial implementation for `Refresh()` method of `CloudProvider` interface done

Usage of the deprecated injection mechanisms in controller-runtime (like `InjectScheme`, `InjectLogger`, `InjectConfig`, `InjectClient`, `InjectCache` etc) as well as package `extensions/pkg/controller/common` are dropped in a preparation to upgrade to the next version where injection is removed entirely. With this, `Inject*` functions on controllers, predicates, actuators, delegates, and friends are not called anymore. When upgrading the `gardener/gardener` dependency to this version, all injection implementations need to be removed. As a replacement, you can get the needed clients and similar from the manager during initialisation of the component.

The `core/v1alpha1` API version is dropped. Before upgrading to this version, make sure that there are no resources in the etcd stored in the `core/v1alpha1` API version. Otherwise, the gardener-apiserver@v1.72.0 will fail to start.

Update alpine base image components to 3.18.3.

Fixes a bug in backup-restore which falsely detects scale-up scenario incase of rolling update of statefulset.

Update etcd-custom-image to `v3.4.26-2`.

`gardener-operator` now takes over management of `fluent-operator` and `vali`.

To support workerless Shoots, extensions reconciling `extensions.gardener.cloud/v1alpha1.Extension` resources need to make adaptions if needed and then set `spec.resources[].workerlessSupported` to `true` in the `ControllerRegistration` for their respective extension type.

The deprecated `allow-to-seed-apiserver` `NetworkPolicy` is no longer available in garden or seed clusters. Use `allow-to-runtime-apiserver` instead.

Shoot fields `.spec.dns.providers[].domains` and `.spec.dns.providers[].zones` are now deprecated and expected to be removed in version `v1.87`. Please plan ahead to drop using those fields in extensions.

Configuring multiple `reserve-excess-capacity` deployments on `Seed`s is supported now by specifying `.spec.settings.excessCapacityReservation.configs`.

The `--node-monitor-grace-period` flag of `kube-controller-manager` is now defaulted to `40s` for Shoot clusters using Kubernetes version `1.27` and higher.

The `check-apidiff` check was changed to only report incompatible and critical changes which need inspection from the developer's side.

A bug causing the gardenlet to panic when a ETCD encryption key rotation operation is triggered for a hibernated Shoot is now fixed. Now, triggering ETCD encryption key rotation or ServiceAccount signing key rotation is forbidden when the Shoot is in waking up phase.

The `gardenlet` and `gardener-operator` Helm charts allow to define toleration seconds for `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`. This configuration considered for their own Deployment as well as the Gardenlet's or Operator's config. The values are set to `60s` by default.

The deprecated `extensions/pkg/controller/worker.{Options,ApplyMachineResources{ForConfig}}` symbols have been dropped since `gardenlet` takes over management of the `machine.gardener.cloud/v1alpha1` API CRDs since `gardener/gardener@v1.73`.

Updated kubernetes dependencies from `1.25.0` to `1.26.2`

An optional field `workerlessSupported` is added under `spec.resources` in the  `ControllerRegistration` API.

The following images are updated:
- `registry.k8s.io/kube-state-metrics/kube-state-metrics`: `v2.5.0` -> `v2.8.2`

status.Status now captures underline cause, allowing consumers to introspect the error returned by the provider. WrapError() function could be used to wrap the provider error

Gardener base image is updated to `gcr.io/distroless/static-debian12:nonroot`.

A bug preventing `plutono` ingress to use `wildcard-certificate` is fixed.

Operators can now view and manage dashboards for compaction jobs running in shoot control plane.

The field `.spec.secretRef` in the `Seed` API has been deprecated and will be removed in a future release of Gardener.

A bug that prevented referencing `ConfigMap`s in `.spec.resources` in `Shoot`s has been fixed.

The plutono dashboards are now verified as part of `make check`.

An issue has been fixed which might have caused the deletion of `Shoot` clusters to stuck when a namespace was forcefully removed before all relevant resources have been cleaned up.

Added `errorCode` field in the `LastOperation` struct. This should be implemented only for the `CreateMachine` call in the `triggerCreationFlow`. This field will be utilized by Cluster autoscaler to do early backoff

`maintenance-controller` now disables `PodSecurityPolicy` admission controller when forcefully upgrading the Kubernetes version of a `Shoot` to `v1.25`. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades to `v1.27`.

The static token kubeconfig can no longer be enabled for Shoot clusters using Kubernetes version `1.27` and higher.

When deploying this version of `gardener-operator`, make sure that you update your `Garden` resources with the new `.spec.virtualCluster.gardener.clusterIdentity` field. If you already have a `gardener-apiserver` deployment, make sure that the value matches the `--cluster-identity` flag of the current `gardener-apiserver` deployment.

The logging e2e event logger test is now adapted to vali logging stack.

Etcd-druid will now deploy distroless `etcd-wrapper` and `etcd-backup-restore` images. Please refer to [etcd-wrapper](https://github.com/gardener/etcd-wrapper) for more information.

The `gardener.cloud/timestamp` annotation is now formatted as `time.RFC3339Nano`.

Added a safety check before adding a learner(non-voting) member in etcd cluster.

Several default settings of Kubernetes feature gates have been corrected.

`gardener-operator` is now managing the `gardener-resource-manager` instance as part of the virtual garden cluster control plane. It provides a `TokenRequest` API-based kubeconfig for `gardener-operator` to access the virtual garden cluster. The static token kubeconfig is now unconditionally disabled.

A bug has been fixed in the [HighAvailabilityConfig-Webhook](https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md#high-availability-config) which caused duplicated entries for zone affinities.

The `pkg/utils/gardener.IntStrPtrFromInt` function has been renamed to `IntStrPtrFromInt32` since `intstr.FromInt` is deprecated.

The `Worker` state reconciler has been dropped, i.e., updated provider extensions will no longer populate the machine state to the `.status.state` field of `Worker` resources. For a few releases, `gardenlet` will no longer persist any still existing data in the `.status.state` field of `Worker` resources during a control plane migration of a `Shoot`, and it will set `.status.state` to `nil` after a successful reconciliation or restore operation.

As of Kubernetes `v1.27`, Gardener enforces a `worker.maximum` configuration for system component worker pools. The value must be greater or equal to the number of zones configured for this pool. This ensures, that the pool has the minimum required nodes to schedule system component across nodes.

Added a LeaderElectionID to the controller options, allowing to run multiple instances of HVPA with leader election when `--leader-elect=true` is passed as commandline arg

All `fluent-bit`-related configuration options have been removed from `gardenlet`'s component configuration.

Update `vertical-pod-autoscaler` to `v0.14.0`.

Gardener now reports `node`s for which the `checksum/cloud-config-data` hasn't been populated yet. This could point towards an error on the node and that not all Gardener related configuration happened successfully.

gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed.

Backupbucket/backupentry controllers: watch secret metadata only

A new optional constraint `CRDsWithProblematicConversionWebhooks` is introduced in the `Shoot` status. This constraint indicates that there is at least one CRD in the cluster which has multiple stored versions and a conversion webhook configured, which could break the reconciliation flow of a `Shoot` in some cases.

Bump builder image golang from `1.20.4` to `1.20.6`

The following mapper funcs from the extension library no longer accept a `context.Context` arg - `ClusterToContainerResourceMapper`, `ClusterToControlPlaneMapper`, `ClusterToDNSRecordMapper`, `ClusterToExtensionMapper`, `ClusterToInfrastructureMapper`, `ClusterToNetworkMapper`, `ClusterToWorkerMapper` and `ClusterToObjectMapper`. The `context.Context` arg was redundant and not used.

The flags which went out-of-support in MCM v0.49.0 have been cleaned up from MCM deployment yaml.

Plutono is now updated to v7.5.22

The `gardener-scheduler` now populates scheduling failure reasons to the `Shoot`'s `.status.lastOperation.description` field.

A new make target is introduced to add license headers.

It is now possible to specify custom linux kernel settings per worker pool for `Shoot`s via `.spec.provider.workers[].sysctls`, which may override Gardener default values.

It is possible now to trigger a seed reconciliation by annotating the Seed with `gardener.cloud/operation=reconcile`.

The following image is updated:
- quay.io/brancz/kube-rbac-proxy: v0.14.0 -> v0.14.2

A bug has been fixed that prevented users without permissions to list `CustomResourceDefinition`s from interacting with the Gardener APIs when using a `kubectl` version lower than `1.27`.

`gardener-operator` is now managing the `nginx-ingress-controller` and `nginx-ingress-k8s-backend` components. Make sure that your `Garden` resource specifies the [`.spec.runtimeCluster.ingress` section](https://github.com/gardener/gardener/blob/ee3dd5d177be1bf3435534f194e25cef67177650/example/operator/20-garden.yaml#L16-L22).

Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance.

`nginx-ingress-controller` image is updated to `v1.9.1`.

The `spec.secretBindingName`, `.spec.networking`, `.spec.networking.type`, `spec.maintenance.autoUpdate.machineImageVersion` fields in the Shoot API are now made optional to prepare for the introduction of workerless Shoots feature. Please see https://github.com/gardener/gardener/issues/7635 for more details.

The `kube-apiserver` no longer mounts root CA bundles from the underlying host.

Gardener sets [`minDomains`](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#spread-constraint-definition) for shoot system components to the number of zones configured in the system component worker pool(s). 
⚠️ It is strongly recommended to place at least one worker node per availability zone for system component workers in order to ensure hitch-free rolling updates and scheduling of pods. You may need to adjust the `maximum: <number-of-zones>` values of your system component worker pool(s).
This configuration only takes effect for clusters which enabled feature gate `MinDomainsInPodTopologySpread` (enabled by default as of Kubernetes `v1.27`).

Fix network annotations to allow fluent-bit connecting to shoot Valis.

`gardener-resource-manager`'s `system-components-config` webhook no longer adds the toleration for the `ToBeDeletedByClusterAutoscaler` taint to system components in shoot clusters. The `ToBeDeletedByClusterAutoscaler` taint is maintained by the `cluster-autoscaler`. This was breaking `cluster-autoscaler`'s drain mechanism when scaling down an under-utilized node. It was causing just evicted system components from to be deleted node to be scheduled again on the to be deleted node.

Base alpine image upgraded from `3.15.7` to `3.15.8`

Bump alpine base version for Docker build to `3.18.2`.

Feature gates have been introduced in etcd-druid, and can be specified using CLI flag `--feature-gate`.

It is no longer possible to configure `.spec.virtualCluster.kubernetes.kubeAPIServer.authorization` in the `Garden` API.

Refactored `statefulset`, `service`, `poddisruptionbudget`, `lease`, and `configmap` components to use default labels and owner references from `etcd`.

The `networking.resources.gardener.cloud/from-policy-pod-label-selector` and `networking.resources.gardener.cloud/from-policy-allowed-ports` annotations are now deprecated and will be removed in the future. Use `networking.resources.gardener.cloud/from-<pod-label-selector>-allowed-ports=<ports>` instead.

`gardener.cloud/operation` annotation was introduced to `seeds`. This includes a verification of its value. Please check your `seeds` for this annotation and remove it if necessary prior to the update.

The `WorkerlessShoots` feature gate has been promoted to beta and is now turned on by default. Before deploying this Gardener version, make sure that all your registered extensions support this feature gate.

Introduce DEPs (Druid Enhancement Proposals) for proposing large design changes in etcd-druid.

Gardener refined the scope of the problematic webhook matcher for `endpoint` objects. Earlier, shoot clusters were assigned a constraint reporting a problem with a `failurePolocy: Fail` webhook acting on these objects. Now, only `endpoint`s in the `kube-system` and `defaults` namespaces are considered for this check.

Two additional labels `worker.gardener.cloud/image-name` and `worker.gardener.cloud/image-version` are attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment.

Resolved an issue where the Custodian Controller was not updating the `Replicas` field in the `etcd` status to reflect the `CurrentReplicas` from the StatefulSet status. This fix ensures consistent behavior with the `etcd` Controller in Druid.

`gardener-resource-manager` now disables cache only for `Secrets` and `ConfigMap` if `DisableCachedClient` set to true.

Update gardener/gardener to 1.77.1.

Add `Care` reconciler to `Garden` controller in `gardener-operator`.

Fix verification.

The two additional labels `worker.gardener.cloud/image-name` and `worker.gardener.cloud/image-version` that were previously introduced and attached to worker nodes are removed again to fix a regression that causes the `kubelet` to restart on nodes that are due to be upgraded to a new OS but not rolled yet which causes their `Pod`s to become temporarily unready.

Go version is updated to 1.20.5.

The GA-ed feature gates `HAControlPlanes` and `FullNetworkPoliciesInRuntimeCluster` have been removed.

The `virtual-garden-kube-apiserver` service (for the `virtual-garden` cluster) was switched from type `LoadBalancer` to `ClusterIP`. Please make sure to migrate all DNS records from the `virtual-garden-kube-apiserver` to the `istio-ingressgateway` endpoint before upgrading to this Gardener version.

When `Shoot`s were updated from non high-availability to `zone` high-availability, it could happen that the control-plane was scheduled to two instead of three zones. This issue is relevant for cloud providers with an inconsistent zone naming (`Azure` is currently the only candidate to our knowledge).
Existing shoots with the before mentioned problem must be fixed manually be operators if required. An automatic move of `etcd`s and their volumes is not part of this fix due to availability reasons.

The `extensions/pkg/controller/operatingsystemconfig/oscommon` package is deprecated and will be removed as soon as the `UseGardenerNodeAgent` feature gate has been promoted to GA. OS extension developers should start adapting to this new feature, see [documentation](https://github.com/gardener/gardener/blob/master/docs/extensions/operatingsystemconfig.md#what-needs-to-be-implemented-to-support-a-new-operating-system) and [example](https://github.com/gardener/gardener/tree/master/pkg/provider-local/controller/operatingsystemconfig) based on `provider-local`.

Following dependencies are updated:
  Go - 1.20.3 
  client-go - v0.26.2 
  controller-runtime - v0.14.5
  gomega - v1.27.1
  zap - v1.24.0 
  gardener/gardener v1.69.0
  k8s (api and apimachinery) - v0.26.2

The `MachineControllerManagerDeployment` has been promoted to beta and is now enabled by default. Make sure that all registered provider extensions support this feature gate before upgrading to this version of Gardener.

`gardener-operator` now deploys `Istio` components into the garden runtime cluster.

Backup-restore waits for its etcd to be ready before attempting to update peerUrl

When the Kubernetes control plane version is at least `v1.28`, it is now possible to set the worker pool Kubernetes version to be at most three versions behind the control plane version. Earlier, only a skew of at most two versions was allowed. Find more details [here](https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#changes-to-supported-skew-between-control-plane-and-node-versions).

Introduce `Spec.Backup.DeltaSnapshotRetentionPeriod` in the `Etcd` resource to allow configuring retention period for delta snapshots.

Update golang 1.19.5 -> 1.20.4

`nginx-ingress-controller-seed` image is updated to `v1.8.0` for `1.24.x+` seeds.

Etcd-backup-restore now uses the user home directory to create files.

Kubernetes feature gate `UnauthenticatedHTTP2DOSMitigation` is considered valid for versions >= `1.25`.

A bug has been fixed which prevented components using the `networking.resources.gardener.cloud/from-world-to-ports` annotation from being reached from internal IP addresses when the cluster was using Cilium as CNI.



Release notes were shortened since they exceeded the maximum length allowed for a pull request body. The remaining release notes will be added as comments to this PR.

.com/gardener/gardener #8640 @oliver-goetz maxSurge for kube-apiserver and gardener-apiserver of the virtual garden cluster is set to 100%.

```bugfix user github.com/gardener/gardener #8122 @timuthy
A regression was fixed that prevented deletions for shoot clusters which were created with a wrong configuration (e.g. with an unavailable domain name).

Removed dead metrics code and refactored the remaining metrics code

`nginx-ingress-controller-seed` image is updated to `v1.7.1` for `1.24.x+` seeds.

`gardener-operator` now takes over management of `plutono`.

Druid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance.

The worker count for the [NetworkPolicy controller](https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md#networkpolicy-controller) in GRM was increased to `20`. This is necessary to create and update `NetworkPolicies` in time, esp. on larger seed clusters.

Concurrent empty machines bulk deletion can now be configured for `cluster-autoscaler` via the field `.spec.kubernetes.clusterAutoscaler.maxEmptyBulkDelete` in the `Shoot` API .

It is now possible to configure `.spec.virtualCluster.gardener.gardenerAPIServer.auditWebhook` in the `Garden` API.

Etcd druid will now not support `policy/v1beta1` for `PodDisruptionBudget`s and will only use `policy/v1` for `PodDisruptionBudget`s

Gardener-based e2e test for the event-logger.

unit tests framework introduced to test implemented methods of `Cloudprovider` and `Nodegroup` interface

An issue causing several tasks from the Shoot reconciliation flow to fail with transient errors of type `duplicate filename in registry` is now fixed.

Etcd-backup-restore now uses a distroless image as its base image. It is no longer compatible with [etcd-custom-image](https://github.com/gardener/etcd-custom-image), and must be used with [etcd-wrapper](https://github.com/gardener/etcd-wrapper) instead.

The no longer required `--gardenlet-manages-mcm` option has been removed. All code in provider extensions related to management/deployment of `machine-controller-manager` should be removed.

While scaling up a non-HA etcd cluster to HA skipping the scale-up checks for first member of etcd cluster as first member can never be a part of scale-up scenarios.

extension library: State update for a Worker object can be now skipped by annotating it with `worker.gardener.cloud/skip-state-update=true`.

Bumped up the custom image version to v3.4.13-bootstrap-11

The `gardener-operator` now enables full `NetworkPolicy` protection for the garden cluster. In case your garden cluster is a seed at the same time, make sure to keep the values of the `FullNetworkPoliciesInRuntimeCluster` feature gate in sync for both `gardener-operator` and `gardenlet`.

`gardenlet` no longer reports the `Bootstrapped` condition on `Seed`s. Instead, it now reports the progress in `.status.lastOperation`, similar to how it's done for `Shoot`s.

Eliminated `RoleBinding` helm charts and converted into Golang component with added unit tests.

Shoot addon `nginx-ingress-controller` image is updated to `v1.3.0` for `v1.22+` shoots.

When Seed's `spec.settings.ownerChecks.enabled=false` gardenlet is now able to delete the owner DNSRecord for a Shoot stuck in deletion where the kube-apiserver Deployment is missing but the Infrastructure is present and cannot be deleted for some reason (infrastructure dependency, invalid credentials).

 Fix an issue, where DNS lookups for non-existing pods of a StatefulSet yielded one of the existing pods even when it should not have.

Vali is now updated to version v2.2.6

The following Golang dependencies have been updated:
- `k8s.io/*` from `v0.28.2` to `v0.28.3`
- `sigs.k8s.io/controller-runtime` from `v0.16.2` to `v0.16.3`

Add new flag `metrics-scrape-wait-duration` for compaction controller to set a wait duration at the end of every compaction job, to allow for metrics to be scraped by a Prometheus instance.

The shoot namespace in seeds is redeployed during shoot deletion to update the zones in use.

The [`highavailabilityconfig` webhook](https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md#high-availability-config) configures topology spread constraints with `minDomains=<number-of- zones>`. This configuration only takes effect for clusters which enabled feature gate `MinDomainsInPodTopologySpread` (default as of Kubernetes `v1.27`). Please note, this configuration will require at least one worker node per registered availability zone so that Kubernetes can spread the respective seed, shoot and control-plane pods across zones.

`fluent-operator` is now installed in the `garden` namespace of seed clusters and will take care of the entire lifecycle of the `fluent-bit` `DaemonSet`.

The deprecated feature gate `APIServerSNI` has been removed.

Adding Gardener-managed finalizers (e.g., `gardener` or `gardener.cloud/reference-protection`) to the `Shoot` on creation is now forbidden.

If you are using `provider-extension` setup you should adapt your files in `example/provider-extensions/garden/controlplane` because `default-domain` and `internal-domain` secrets are removed from `gardener-controlplane` Helm chart.

A new alpha feature gate `DisableScalingClassesForShoots` has been introduced on `gardenlet`. If turned on, initial resource requests for `kube-apiserver`s of shoot clusters running on seed clusters which enable the `HVPA` feature gate are assigned statically and no longer by a scaling class determined by maximum node count. This helps to reduce resource waste for clusters with little usage.

Update golang base container image to 1.21.0.

`gardener-operator` now runs a new controller which protects `Secret`s and `ConfigMap`s with a finalizer in case they are referenced in `Garden` resources.

The `lastUpdateTime` of extension conditions is no longer considered. Ensure that all registered extensions populate the `lastHeartbeatTime` field instead.

A bug is fixed that rendered the "CPU usage" panel of the "VPN" Plutono dashboard blank.

:warning: `etcd.Status.ClusterSize`, `etcd.Status.ServiceName`, `etcd.Status.UpdatedReplicas` have been marked as deprecated and users should refrain from depending on these fields.

Run `make ci-e2e-kind` to run the e2e tests on local machine

Introduced `delta-snapshot-retention-period` CLI flag to extend the configurable retention period for delta snapshots in `etcd-backup-restore`, enhancing flexibility for backup retention.

Gardener autoscaler now backs-off early from a node-group (i.e. machinedeployment) in case of `ResourceExhausted` error. Refer docs at `https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/FAQ.md#when-does-autoscaler-backs-off-early-from-a-node-group` for details.

The admission controllers of common provider extensions are automatically installed in the local extensions development setup

Add new grafana dashboard of seed deployment replicas

Deactivate leader election, health and readiness checks when running `make *-debug.`

The local deployment of Gardener is extended so that it is now possible to create a second single zone HA `Seed`. This `Seed` can be used to test the control plane migration scenario for HA `Shoot`s. Additionally, make targets were added to trigger the control plane migration integration test with HA `Shoot`s: `test-e2e-local-migration-ha-single-zone` to test the migration locally, and `ci-e2e-kind-migration-ha-single-zone` mainly intended to be used in Gardener prow jobs.

The target cache for `gardener-resource-manager` is now unconditionally enabled, leading to faster reconciliations and less network I/O.

`machinepriority.machine.sapcloud.io` annotation on machine is now reset to 3 by autoscaler if the corresponding node doesn't have `ToBeDeletedByClusterAutoscaler` taint

This PR aligns container build targets with project CI supporting multi-platform builds and simplifies overall Makefile structure.

⚠️ The deprecated field `.spec.kubernetes.kubeAPIServer.enableBasicAuthentication` has been removed from the Shoot API. Please check your `Shoot`s manifests and remove the `.spec.kubernetes.kubeAPIServer.enableBasicAuthentication` field.

Extensions have to implement the `ForceDelete` function in the actuator with the logic of forcefully deleting all the resources deployed by them.

The GA-ed feature gates `SeedChange` and `CopyEtcdBackupsDuringControlPlaneMigration` have been removed.

Extensions running on seed clusters can get access to the garden cluster by using the injected kubeconfig specified by the `GARDEN_KUBECONFIG` environment variable. You can read about the details in this [doc](https://github.com/gardener/gardener/blob/master/docs/extensions/garden-api-access.md).

The `{github.com/gardener/gardener/pkg/apis/core/helper,github.com/gardener/gardener/pkg/apis/core/v1beta1/helper}.SeedSettingOwnerChecksEnabled` will now return `false` if the corresponding Seed setting is `nil`. Previously, the func was returning `true` when the Seed setting is `nil`.

A bug has been fixed in the `garden/fluent-bit` that caused a failure in creating `networkpolicies` for scraping metrics.

Introduce flag `metrics-scrape-wait-duration` to `etcdbrctl compact` command, that specifies a wait duration at the end of a snapshot compaction, to allow Prometheus to scrape metrics related to compaction before the `etcdbrctl` process exits.

The `pkg/utils/secrets` package now signs certificates with 3072 bit RSA keys.

Deprecated annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns` is removed. Use field `.spec.systemComponents.nodeLocalDNS.enabled` in `Shoot` instead. Switching on node-local-dns via shoot specification will roll the nodes even if node-local-dns was enabled beforehand via annotation.

`UseEtcdWrapper` feature gate has been introduced to allow users to opt for the new [etcd-wrapper](https://github.com/gardener/etcd-wrapper) image.

Extensions that wish to be scraped by the `seed-prometheus` must annotate their pods with `prometheus.io/scrape=true` along with `prometheus.io/name=<name>`. See https://github.com/gardener/gardener/blob/master/docs/monitoring/README.md#seed-prometheus for more details.

Upgrade to go 1.20.3.

Update base image of `ingress-default-backend` to alpine:3.18.3

The following images are updated:
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.21.5` -> `v1.21.6` (for Kubernetes `1.21`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.22.5` -> `v1.22.6` (for Kubernetes `1.22`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.23.3` -> `v1.23.4` (for Kubernetes `1.23`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.24.2` -> `v1.24.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.25.2` -> `v1.25.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.26.1` -> `v1.26.2` (for Kubernetes `1.26`)

Update golang 1.19.5 -> 1.20.4

The unused `github.com/gardener/gardener/pkg/controllerutils/predicate.IsBeingMigratedPredicate`, `github.com/gardener/gardener/pkg/controllerutils/predicate.IsObjectBeingMigrated` and `github.com/gardener/gardener/pkg/utils/gardener.IsObjectBeingMigrated` funcs are now removed.

A bug has been fixed which could cause `kube-proxy`s from being missing after a `Shoot` has been woken up from hibernation.

The `extensions/pkg/controller.Use{TokenRequestor,ServiceAccountTokenVolumeProjection}` functions have been removed since they always return `true`.

The shoot namespace in seeds is redeployed during the shoot migration flow to update the zones in use.

Probes will not be created for shoots with no workers.

`extensions.gardener.cloud/v1alpha1.ControlPlane` is now deployed after `kube-apiserver` in the Shoot reconciliation flow.

So far the `github.com/gardener/gardener/pkg/utils/managedresources.{NewForShoot,CreateForShoot}` funcs were ignoring the passed `origin` func parameter and were always using `gardener` as value. These funcs will now respect and use the passed `origin` value.

Now the vali ingress definition points to the shoot logging service.

Bump builder image golang from `1.19.5` to `1.20.2`

`Shoot`s allow to optionally configure a specific scheduler via `.spec.schedulerName`. The `default-scheduler` is used in case non is configured. Please note, that `Shoot`s will remain `Pending` in case a scheduler name is configured but an adequate scheduler is not available in the landscape.

A bug is fixed in the Prometheus alert definitions that caused false positive KubePodNotReadyControlPlane alerts related to the etcd compaction job.

Deprecated annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns-force-tcp-to-{cluster-dns, upstream-dns}` is removed. Use field `.spec.systemComponents.nodeLocalDNS.{forceTCPToClusterDNS, forceTCPToUpstreamDNS}` in `Shoot` instead.

Updated to go v1.20.5

Improves client recreate during cluster reconcile.

⚠️ Gardener does no longer support garden, seed, or shoot clusters with Kubernetes versions < 1.22. Make sure to upgrade all existing clusters before upgrading to this Gardener version.

Annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns-force-tcp-to-{cluster-dns, upstream-dns}` is deprecated and will be removed in future releases. Use field `.spec.systemComponents.nodeLocalDNS.{forceTCPToClusterDNS, forceTCPToUpstreamDNS}` in `Shoot` instead.

`default-domain`, `internal-domain`, `alerting` and `openvpn-diffie-hellman` secrets are removed from `gardener-controlplane` Helm chart. Please ensure to update them in a different way before upgrading Gardener. If you would like to prevent Helm from deleting these secret during the upgrade, you could annotate them with `"helm.sh/resource-policy": keep`.

Go version is updated to 1.20.6.

The following image is updated:
- `quay.io/prometheus/alertmanager`: `v0.24.0` -> `v0.26.0`

Block public access for S3 buckets created by integration tests.

A bug causing unnecessary reorder of extension in `Shoot` `spec.extensions` is fixed.

Status of `garden` now includes the `ObservabilityComponentsHealthy` condition which show the health of observability components in the garden runtime-cluster.

⚠️ The deprecated field `.spec.settings.ownerChecks` has been removed from the Seed API. Please check your `Seed`s and remove any usage before upgrading to this Gardener version.

All components in the gardener logging stack are now updated to the following respective versions. Fluent-bit to 2.1.4, Fluent-operator to 2.3.0 and logging to 0.55.3

A bug is fixed that prevented scraping the metrics of etcd in the shoot control plane.

An issue causing deletion of a legacy (wrongly configured) Shoot cluster to be denied because of network ranges overlapping with the default VPN network is now fixed.

Gardener now uses 3072 bit RSA keys in order to generate TLS certificates.

`leader-election-resource-lock` flag is dropped and the leader-election resource-lock is hard coded to leases.

Using internal API versions in `providerConfig` fields is no longer permitted (deprecated since more than `2y`). Ensure that you always use a versioned API.

`nginx-ingress-controller` image is updated to `v1.8.1` for Kubernetes`v1.24+` clusters.

Annotations in `seed.spec.settings.loadBalancerServices.annotations` are now applied to the Nginx-Ingress load balancer service in the seed cluster.

A new alpha feature gate named `MachineControllerManagerDeployment` has been introduced in `gardenlet`. Only enable it when all registered provider extensions in your landscape support this feature.

For Shoot clusters using Kubernetes version `1.27` and higher, the `.spec.kubernetes.kubeControllerManager.podEvictionTimeout` field has no effect anymore since the backing `--pod-eviction-timeout` CLI flag has been removed.

Updated go to 1.19.9

The `fluent-bit-vali-plugin` now supports fluent-bit v2.1.0 and above.

Provider extensions must now pass the `cluster.Cluster` object for the garden cluster to the `genericactuator.NewActuator` function. See [this](https://github.com/gardener/gardener/blob/8d2f116aa606e5181cd430e5063dd798629bdc78/cmd/gardener-extension-provider-local/app/app.go#L228-L246) for an example how to create such a `cluster.Cluster` object.

An issue causing nil pointer panic on scaleup of the machinedeployment along with trigger of rolling update, is fixed

The `Deploying Shoot namespace in Seed` step was slightly improved. Earlier it failed at some occasions when it tried to read zone information for volumes that have not been created yet. This was a transient error that dissolved in subsequent reconcile runs.

Gardenlet can now set feature gates for `etcd-druid`. They can be specified via the gardenlet configuration `GardenletConfiguration.EtcdConfig.FeatureGates`

The `ResourcesProgressing` condition appearing in the status of `ManagedResource`s now checks for non-terminated `Pod`s before reporting `status=False`.

Prometheus scrape job configs for targets in the shoot cluster have been improved.

update client-go version and exclude the old one in go.mod

Etcd snapshot compaction jobs will now be named `<etcd-name>-compactor` for better readability for human operators.

Revendors the bbolt from `v1.3.6` to `v1.3.7`

`gardener-operator` configures SNI components in order to expose the `virtual-garden-kube-apiserver` via the `istio-ingressgateway` in the Garden cluster.
With this change, operators can start to switch DNS records from the `virtual-garden-kube-apiserver` service to the `istio-ingress` service endpoint. The type of the `virtual-garden-kube-apiserver` service will soon be switched from `LoadBalancer` to `ClusterIP`.

`AllMembersReady` condition has now been fixed to eventually show the correct overall readiness of an etcd cluster.

gardenlet: A regression causing metering related recording rules for the aggregate-prometheus not to be applied is now fixed.

Introduces a skaffold local development pipeline to fluent-bit-vali-plugin

Provider extensions should be adapted such that they only inject their provider-specific `machine-controller-manager` sidecar container into the `machine-controller-manager` deployment instead of managing the full deployment themselves. In the future, `gardenlet` will take over managing it. Please see https://github.com/gardener/gardener/pull/8019 for an example how `provider-local` was adapted and replicate it for your provider extensions.

Add an alert for VPNHAShootNoPods when shoot in HA (high availability) mode.

The following dependencies are updated:
- `k8s.io/*` : `v0.26.4` -> `v0.27.5`
- `sigs.k8s.io/controller-runtime`: `v0.14.6` -> `v0.15.2`

The logging components: vali and valitail are now updated to v2.2.8.

HVPA supports k8s versions >= 1.25 by switching to `k8s.io/autoscaling/v2` when necessary for all API calls.

Gardener Scheduler's Minimal Distance strategy can take scheduling decisions based on region distances configured by operators. This especially improves the allocation for shoots of providers regions for which the standard Levenshtein distance is inappropriate. Please see `docs/concepts/scheduler.md` for more information.

Makefile targets have changed: Introduced gardener-setup, gardener-restore, gardener-local-mcm-up, non-gardener-setup, non-gardener-restore,  non-gardener-local-mcm-up. Users can also directly use the scripts which are used by these makefile targets.

Package `pkg/utils/managedresources` now works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource.

It is possible now to create a workerless shoot cluster when the `WorkerlessShoots` feature gate in the `gardener-apiserver` is enabled. Please see [this document](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_workerless.md) for more details.

Prevent fluent-bit-to-vali plugin panic when Cluster is updated and its Shoot has no lastOperation set

Change `log` mount path of `node-problem-detector` from `/var/log` to `/var/log/journal`.

The `DisablingScalingClassesForShoots` feature gate has been promoted to beta.

The `.{source,target}ClientConnection.namespace` field has been renamed to `namespaces` and now takes a list of namespaces. The `.targetClientConnection.disableCachedClient` field has been removed.

The `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler` image has been updated from `v1.26.2` to `v1.27.0` (for Kubernetes `>= 1.27`).

Feature gate `APIServerFastRollout` for `gardenlet` is introduced and enabled by default. When enabled, `maxSurge` for `kube-apiservers` of `Shoot`s is set to `100%`.

Introduce DEP-04 [EtcdMember Custom Resource](https://github.com/gardener/etcd-druid/blob/master/docs/proposals/04-etcd-member-custom-resource.md).

When the `ShootForceDeletion` featuregate in the apiserver is turned on, users will be able to force-delete the Shoot. You **MUST** ensure that all the resources created in the IaaS account are cleaned up to prevent orphaned resources. Gardener will **NOT** delete any resources in the Shoot cloud-provider account. See [Shoot Force Deletion](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_operations.md#force-deletion) for more details.

Applying Gardener resources server-side has caused the `the server is currently unable to handle the request` error which is now fixed.

Removed apiserver-proxy pod webhook as it is now included in Gardener Resource Manager.

Since `Namespace`s are no longer deleted (and forcefully finalized after some grace period), the `shoot.gardener.cloud/cleanup-namespaces-finalize-grace-period-seconds` annotation does no longer have any effect. Relevant Kubernetes resources are still cleaned up (see [this document](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_cleanup.md)) for more information.

The deprecated `.spec.virtualCluster.dns.domain` field has been dropped from the `Garden` API. Make use of `.spec.virtualCluster.dns.domains`.

Force drain and delete volume attachments for nodes un-healthy due to `ReadOnlyFileSystem` and `NotReady` for too long

Grafana and Loki are replaced with the fork of their last Apache 2.0 licensed releases: Plutono and Vali, that will continue to receive security updates.

Bump g/g version to remove stale client-go dependency

The `HAControlPlanes` feature gate has been promoted to beta and is now turned on by default.

The project vendors the latest released gardener version - v1.73.0

Gardener now allows to omit or to only partially define Kubernetes versions in `Shoot`s. The version will automatically be defaulted to the latest minor and/or patch version found in the linked `CloudProfile`.

`hack/generate.sh` has been renamed to `hack/generate-sequential.sh`.

If the `kubeletCSRApprover` controller is enabled, it is now mandatory to specify the namespace in the source cluster in which the `Machine` resources reside via `.controllers.kubeletCSRApprover.machineNamespace`.

etcd-custom-image updates from `v3.4.13-bootstrap-9` to `v3.4.13-bootstrap-10`

The following image is updated:
- `quay.io/prometheus/prometheus`: `v2.43.1` -> `v2.47.0`

A new controller in `gardenlet` for periodically backing up the `ShootState` for `Shoot`s has been introduced. This controller is only activated when `gardenlet` is responsible for an unmanaged `Seed` (i.e., one not backed by a `ManagedSeed` object). By default, backups are taken roughly each `6h`.

A `generate-admin-kubeconf.sh` script which can be used to generate an admin kubeconfig for a local shoot cluster was added in the `hack/usage` directory.

`uncachedObjects` under pkg/client/kubernetes/options.go is now removed from Config struct which is used to set options for new ClientSets. Now the uncached objects can be directly set under `clientOptions.Cache.DisableFor` field.

Several low timeouts (30s) that were introduced in v1.71.0 for several steps are now reverted as in some cases the Network/ControlPlane reconciliation cannot succeed for 30s.

 Allow the kubelet configuration to define swap behaviour {LimitedSwap / UnlimitedSwap} for k8s >= 1.22

The `register-kind2-env` and `tear-down-kind2-env` will no longer try to deploy and delete the `seed-local` `Secret`. This fixes an issue where `tear-down-kind2-env` would hang as it deletes and then waits for the `seed-local` `Secret` to be deleted which can not happen as long as the `local` `Seed` which uses it still exists.

A bug causing incorrect volume mount path for `Etcd`s and `EtcdCopyBackupsTask`s using `Local` snapshot storage provider while using distroless etcd-backup-restore image `v0.25.x` has been resolved.

Provider extensions should be adapted such that they no longer perform health checks specific to the `machine-controller-manager` deployment or the machines/nodes. In the future, `gardenlet` will take over performing these checks. Please see https://github.com/gardener/gardener/pull/8019 for an example how `provider-local` was adapted and replicate it for your provider extensions.

An edge case where outdated DesiredReplicas annotation blocked a rolling update is fixed.

The `charts/images.yaml` file was moved to `imagevector/images.yaml`.

The kind cluster used in local setup does now use the new way in containerd to configure registry mirrors.

Added an example for `AdminKubeconfigRequest` via the Python Kubernetes client.

The GA-ed `DisableScalingClassesForShoots` feature gate has been removed.

Methods `SkipIf` and `DoIf` for `TaskFn` have been dropped. A new field `SkipIf` is introduced in `Task`, If set to true the task will be skipped and will also not be reported by the progress reporter.

Any resource with a kind other than `ConfigMap` or `Secret` in `.spec.resources` in `Shoot`s is now forcefully removed. New validation has been introduced to prevent adding other resources in the future.

The regression is now fixed and the control plane logs shall be visible in the Plutono dashboards.

Gardener uses an `InternalSecret` per Shoot for syncing the client CA to the project namespace in the garden cluster (named `<shoot-name>.ca-client`). The `shoots/adminkubeconfig` subresource signs short-lived client certificates by retrieving the CA from the `InternalSecret`.

The deprecated `core.gardener.cloud/apiserver-exposure` label and handling has been dropped.

The `shootstate-extensions` and `shootstate-secret` controllers have been dropped. The `gardenlet`'s component config file should be updated to no longer specify related configuration (`.controllers.{shootSecret,shootStateSync}`).

During the `Migrate` phase of a control plane migration of a `Shoot`, the state is now only persisted after all extension resources have been migrated. Consequently, make sure that you have added all state to the `.status.state` field of the respective extension object when running `Migrate()`.

`github.com/gardener/gardener/pkg/utils/gardener.ShootAccessSecret` was renamed to `AccessSecret`.

A bug has been fixed which was causing the garbage collector in `gardener-resource-manager` to wrongfully collect `Secret`s related to `ManagedResource`s when the source and the target cluster are equal.

Added pod security enforce level `baseline` label to Istio-related namespaces. The `garden` and shoot namespaces have the `privileged` level. For extension namespaces, the new `security.gardener.cloud/pod-security-standard-enforce` annotation on  `ControllerRegistration` resources specifies the level. When set, the `extension` namespace is created with `pod-security.kubernetes.io/enforce` label set to `security.gardener.cloud/pod-security-standard-enforce`'s value.

New metrics introduced: 
- api_request_duration_seconds -> tracks time taken for successful invocation of provider APIs. This metric can be filtered by provider and service.
- driver_request_duration_seconds -> tracks total time taken to successfully complete driver method invocation. This metric can be filtered by provider and operation.
- driver_requests_failed_total -> records total number of failed driver API requests. This metric can be filtered by provider, operations and error_code.

The `.spec.kubernetes.kubeAPIServer.serviceAccountConfig.acceptedIssuers` field of the `Shoot` spec no longer allows duplicate values.

Custodian controller no longer watches leases owned by the etcd resources, thus reducing frequency of etcd status updates and now honouring `custodian-sync-period` value.

The `VerticalPodAutoscaler` resources for `kube-proxy`s is no longer recreated when the Kubernetes patch version of the `Shoot` or the respective worker pools is updated. This ensures updated `kube-proxy`s keep the same CPU/memory resource requirements as before the patch version update. In order to put this change into effect, all existing `VerticalPodAutoscaler`s for `kube-proxy`s are getting recreated.

Add support for `Local` provider for e2e tests.

The feature gates `FullNetworkPolicies` and `HAControlPlanes` have been promoted to GA and are now locked to "unconditionally enabled".

`kubectl get garden` now features additional printer column `Observability` providing information about the Observability components of the runtime cluster.

An issue causing `VPN Seed (CPU| Memory) Usage` dashboards not showing data is now fixed.

`gardenlet`'s `ControllerInstallation` controller now populates the feature gate of `gardenlet` via the Helm values to extensions when they are getting installed. The information is populated via the `.gardener.gardenlet.featureGates` key. It contains a map whose keys are feature gates names and whose values are booleans (depicting the enablement status).

A bug was fixed which was causing existing `Bastion` resources on the garden cluster to not be deleted when `SSHAccess` is disabled on a Shoot cluster.

It is required to have `ControllerRegistrations`s for Kinds `ControlPlane`, `Infrastructure` and `Worker` with the same types used for seeds (`seed.spec.provider.type`). This is already the case if seeds and shoots share the same cloud provider. The seed reconciliation flow waits for the associated `ControllerInstallation` to be ready before continuing rolling out seed system components. It allows Gardener provider extensions to ship components that not only act on shoot control-plane but also on seed system components.

Added a new metric that will allow to get the number of stale (due to unhealthiness) machines  that are getting terminated

The kind clusters are now unified to use `garden.local.gardener.cloud` DNS name in the containerd config when configuring registry mirror hostnames. Previously, to access the pull through registry cache some kind clusters were configured to use `garden.local.gardener.cloud`, others - the Node name of the control plane Node.

Add CVE categorization for etcd-druid.

Base image on `telegraf` and `tune2fs` is upgraded from 3.17.2 to 3.18.0

Webhooks remediator sets the timeoutSeonds to 3 seconds for webhook affecting lease resources in `kube-system` namespace only if there is no objectSelector provided in webhook.

The certificate chains served by `kube-apiserver`s does now include the CA certificates used to sign their server certificates.

`gardener-operator` is now managing the `kube-controller-manager` instance as part of the virtual garden cluster control plane.

Decouple progess update of gardener operator from task flow logic and thereby prevent concurrency bugs.

`gardener-operator` no longer reports the `Reconciled` condition. Instead, it now reports the progress in `.status.lastOperation`, similar to how it's done for `Shoot`s.

Add CVE categorization for etcd-backup-restore.

gardener / garden-setup

[ci:component:github.com/gardener/gardener:v1.69.3->v1.82.1] #1236