gardener-robot-ci-1
commented
1 year ago com/gardener/gardener #7766 @Kristian-ZH An issue causing the garden/grafana Pod to fail to reach to the garden/loki Pod on cilium Seed clusters is now mitigated.
```other user github.com/gardener/gardener #7897 @himanshu-kun
The following images are updated:
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.21.5` -> `v1.21.6` (for Kubernetes `1.21`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.22.5` -> `v1.22.6` (for Kubernetes `1.22`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.23.3` -> `v1.23.4` (for Kubernetes `1.23`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.24.2` -> `v1.24.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.25.2` -> `v1.25.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.26.1` -> `v1.26.2` (for Kubernetes `1.26`)Update golang 1.19.5 -> 1.20.4The `pkg/operation/botanist/component/*` resources have been moved to `pkg/component/*`.`hack/generate.sh` has been renamed to `hack/generate-sequential.sh`.The `DisablingScalingClassesForShoots` feature gate has been promoted to beta.Following dependency has been updated:-
- github.com/gardener/etcd-druid v0.18.1 -> v0.18.4The kind cluster used in local setup does now use the new way in containerd to configure registry mirrors.`Secret`s/`ConfigMap`s referenced in `.spec.resources` of `Shoot`s are now protected with a finalizer to ensure they do not disappear from the system as long as they are still referenced somewhere.The following dependencies are updated:
- `k8s.io/*` : `v0.26.4` -> `v0.27.5`
- `sigs.k8s.io/controller-runtime`: `v0.14.6` -> `v0.15.2``gardenlet` no longer reports the `Bootstrapped` condition on `Seed`s. Instead, it now reports the progress in `.status.lastOperation`, similar to how it's done for `Shoot`s.Annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns-force-tcp-to-{cluster-dns, upstream-dns}` is deprecated and will be removed in future releases. Use field `.spec.systemComponents.nodeLocalDNS.{forceTCPToClusterDNS, forceTCPToUpstreamDNS}` in `Shoot` instead.Status of `garden` now includes the `ObservabilityComponentsHealthy` condition which show the health of observability components in the garden runtime-cluster.All `fluent-bit`-related configuration options have been removed from `gardenlet`'s component configuration.Gardenlet can now set feature gates for `etcd-druid`. They can be specified via the gardenlet configuration `GardenletConfiguration.EtcdConfig.FeatureGates``gardener-operator` now takes over management of `gardener-metrics-exporter`.A guideline for developers regarding [`TODO` statements](https://github.com/gardener/gardener/blob/docs/master/development/process.md#todo-statements) has been introduced.An issue has been fixed that caused traffic from outside of the cluster to `Istio-Ingress` being blocked. This is only relevant if seed(s) specify additional load balancer annotations via `seed.spec.settings.loadBalancerServices.annotations`.It is now possible to perform control plane migration for HA shoot clusters. Grafana and Loki are replaced with the fork of their last Apache 2.0 licensed releases: Plutono and Vali, that will continue to receive security updates.Added a safety check before adding a learner(non-voting) member in etcd cluster.gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed.Two additional labels `worker.gardener.cloud/image-name` and `worker.gardener.cloud/image-version` are attached to worker nodes to identify which operating system they are running. This can then be used in selectors that target only workers with a specific operating system and is helpful for e.g. driver deployment.Etcd-druid will now deploy distroless `etcd-wrapper` and `etcd-backup-restore` images. Please refer to [etcd-wrapper](https://github.com/gardener/etcd-wrapper) for more information.The `spec.secretBindingName`, `.spec.networking`, `.spec.networking.type`, `spec.maintenance.autoUpdate.machineImageVersion` fields in the Shoot API are now made optional to prepare for the introduction of workerless Shoots feature. Please see https://github.com/gardener/gardener/issues/7635 for more details.The `.spec.settings.ownerChecks` field of the Seed configuration is deprecated. The "bad-case" control plane migration is being removed in favor of the HA Shoot control planes (see https://github.com/gardener/gardener/issues/6302). The field will be locked to `false` in a future version of Gardener. In this way gardenlet will clean up all owner DNSRecords. Finally, the field will be removed from the API. Set this field to `false` to be prepared for the above-mentioned locking.The Seed's `.spec.settings.ownerChecks` field is now no-op - the `gardener-apiserver` no longer defaults this field and no longer validates it. The field will be set always to `nil` on CREATE/UPDATE request.
Gardener landscape operators specifying this field should no longer specify it. The field will be removed in a future version of Gardener. Allow the kubelet configuration to define swap behaviour {LimitedSwap / UnlimitedSwap} for k8s >= 1.22Add Prometheus alert for pending seed podsThe following mapper funcs from the extension library no longer accept a `context.Context` arg - `ClusterToContainerResourceMapper`, `ClusterToControlPlaneMapper`, `ClusterToDNSRecordMapper`, `ClusterToExtensionMapper`, `ClusterToInfrastructureMapper`, `ClusterToNetworkMapper`, `ClusterToWorkerMapper` and `ClusterToObjectMapper`. The `context.Context` arg was redundant and not used.Prometheus scrape job configs for targets in the shoot cluster have been improved.Removed `service.beta.kubernetes.io/aws-load-balancer-type: nlb` annotation from istio-ingressgateway service template. Set this annotation in seed configuration. Note: Changing load balancer type creates a new one, old one requires manual clean-up.The deprecated `allow-{to,from}-shoot-apiserver` `NetworkPolicy`s have been dropped. Ensure that all registered extensions have been adapted.Shoot fields `.spec.dns.providers[].domains` and `.spec.dns.providers[].zones` are now deprecated and expected to be removed in version `v1.87`. Please plan ahead to drop using those fields in extensions.Functions `controllerutils.GetAndCreateOrMergePatch`, `controllerutils.GetAndCreateOrStrategicMergePatch`, `controllerutils.CreateOrGetAndMergePatch` and `controllerutils.CreateOrGetAndStrategicMergePatch` were incompatibly changed and now accept a `controllerutils.PatchOption` instead of `client.MergeFromOption`.
If your controllers use one of these functions with `client.MergeFromOption`, you should update it to `controllerutils.PatchOption`.
The `controllerutils.PatchOption` can hold two options today:
- `client.MergeFromOption` which is passed to the underlying patch function.
- `controllerutils.SkipEmptyPatch` which prevents sending empty patches (`{}`).Extensions vendoring this `gardener/gardener` version need to provide RBAC privileges for `PATCH apps/depoyments/scale`.The following images are updated:
- `registry.k8s.io/kube-state-metrics/kube-state-metrics`: `v2.5.0` -> `v2.8.2`Update alpine base image components to 3.18.3.Shoot fields `.spec.dns.providers[].domains` and `.spec.dns.providers[].zones` are now deprecated and expected to be removed in version `v1.87`. Please use the extensions' configuration to configure providers with this ability.`pkg/resourcemanager/controller/garbagecollector/references.InjectAnnotations` now also handles `pods.spec.imagePullSecrets`. If `gardenlet` is responsible for a managed `Seed`, it will delete all `ShootState` resources for its `Shoot`s that are not currently in migration. See also [GEP-22](https://github.com/gardener/gardener/blob/master/docs/proposals/22-improved-usage-of-shootstate-api.md) for further details about the motivation.Gardener denies setting `Shoot.Spec.ControlPlane.HighAvailability.FailureTolerance.Type` if shoot is hibernated.Updated go to 1.20.7Prevent nil pointer exceptions on shoot deletion in `gardenlet` when seed namespace is gone.The field `.spec.secretRef` in the `Seed` API has been deprecated and will be removed in a future release of Gardener.Gardener now allows to omit or to only partially define Kubernetes versions in `Shoot`s. The version will automatically be defaulted to the latest minor and/or patch version found in the linked `CloudProfile`.HVPA supports k8s versions >= 1.25 by switching to `k8s.io/autoscaling/v2` when necessary for all API calls.The `.{source,target}ClientConnection.namespace` field has been renamed to `namespaces` and now takes a list of namespaces. The `.targetClientConnection.disableCachedClient` field has been removed.Deprecated annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns-force-tcp-to-{cluster-dns, upstream-dns}` is removed. Use field `.spec.systemComponents.nodeLocalDNS.{forceTCPToClusterDNS, forceTCPToUpstreamDNS}` in `Shoot` instead.Shoot node network and seed pod network need to be disjoint. This will be checked during scheduling of a shoot cluster, i.e. during initial admission or on control-plane migration.Gardener sets [`minDomains`](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#spread-constraint-definition) for shoot system components to the number of zones configured in the system component worker pool(s).
⚠️ It is strongly recommended to place at least one worker node per availability zone for system component workers in order to ensure hitch-free rolling updates and scheduling of pods. You may need to adjust the `maximum: <number-of-zones>` values of your system component worker pool(s).
This configuration only takes effect for clusters which enabled feature gate `MinDomainsInPodTopologySpread` (enabled by default as of Kubernetes `v1.27`).The `lastUpdateTime` of extension conditions is no longer considered. Ensure that all registered extensions populate the `lastHeartbeatTime` field instead.Since `Namespace`s are no longer deleted (and forcefully finalized after some grace period), the `shoot.gardener.cloud/cleanup-namespaces-finalize-grace-period-seconds` annotation does no longer have any effect. Relevant Kubernetes resources are still cleaned up (see [this document](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_cleanup.md)) for more information.Base alpine image upgraded from `3.15.7` to `3.15.8`The `charts/images.yaml` file was moved to `imagevector/images.yaml`.A bug is fixed that prevented scraping the metrics of etcd in the shoot control plane.Probes will not be created for shoots with no workers.An issue causing `VPN Seed (CPU| Memory) Usage` dashboards not showing data is now fixed.If the `kubeletCSRApprover` controller is enabled, it is now mandatory to specify the namespace in the source cluster in which the `Machine` resources reside via `.controllers.kubeletCSRApprover.machineNamespace`.Shoot addon `nginx-ingress-controller` image is updated to `v1.3.0` for `v1.22+` shoots.Bumped up the custom image version to v3.4.13-bootstrap-11Add failure tolerance option to the `CreateShoot` test.Developer Action Required: The `make deploy` command has been replaced with `make deploy-via-kustomize`. Please update your deployment workflows accordingly.Base alpine image for etcd-custom-image upgraded from `3.15.7` to `3.15.8`Upgrade gardener/gardener from `1.65.0` to `1.76.0`Extensions that wish to be scraped by the `seed-prometheus` must annotate their pods with `prometheus.io/scrape=true` along with `prometheus.io/name=<name>`. See https://github.com/gardener/gardener/blob/master/docs/monitoring/README.md#seed-prometheus for more details.Removed apiserver-proxy pod webhook as it is now included in Gardener Resource Manager.The kind clusters are now unified to use `garden.local.gardener.cloud` DNS name in the containerd config when configuring registry mirror hostnames. Previously, to access the pull through registry cache some kind clusters were configured to use `garden.local.gardener.cloud`, others - the Node name of the control plane Node.A bug preventing `plutono` ingress to use `wildcard-certificate` is fixed.Annotations in `GardenletConfiguration.seedConfig.metadata.annotations` are added to the `Seed` object during registration. If an annotation is removed from `seedConfig`, it is **not** removed from the `Seed` object.The garbage collection controller now also considers managed resources when deciding if secrets/configmaps should be garbage collected.Backupbucket/backupentry controllers: watch secret metadata onlyAny resource with a kind other than `ConfigMap` or `Secret` in `.spec.resources` in `Shoot`s is now forcefully removed. New validation has been introduced to prevent adding other resources in the future.Bump alpine base version for Docker build to `3.18.2`. The `shootstate-extensions` and `shootstate-secret` controllers have been dropped. The `gardenlet`'s component config file should be updated to no longer specify related configuration (`.controllers.{shootSecret,shootStateSync}`).The `virtual-garden-kube-apiserver` service (for the `virtual-garden` cluster) was switched from type `LoadBalancer` to `ClusterIP`. Please make sure to migrate all DNS records from the `virtual-garden-kube-apiserver` to the `istio-ingressgateway` endpoint before upgrading to this Gardener version.It is now easier to annotate `Service`s related to extensions serving webhook handlers that must be reached by `kube-apiserver`s running in separate namespaces such that the respective network traffic gets allowed. Please refer to [this guide](https://github.com/gardener/gardener/blob/master/docs/usage/network_policies.md#webhook-servers) for all information. Extensions serving shoot webhook should make use of this new approach - the old functionality deploying dedicated `NetworkPolicy`s is deprecated and will be removed in the future.To support workerless Shoots, extensions reconciling `extensions.gardener.cloud/v1alpha1.Extension` resources need to make adaptions if needed and then set `spec.resources[].workerlessSupported` to `true` in the `ControllerRegistration` for their respective extension type.`gardener-operator` now deploys `Istio` components into the garden runtime cluster.`operator` now deletes `ManagedResources` deployed to the virtual-garden before deleting `virtual-garden-kube-apiserver`.It is no longer possible to configure `.spec.virtualCluster.kubernetes.kubeAPIServer.authorization` in the `Garden` API.Usage of the deprecated injection mechanisms in controller-runtime (like `InjectScheme`, `InjectLogger`, `InjectConfig`, `InjectClient`, `InjectCache` etc) as well as package `extensions/pkg/controller/common` are dropped in a preparation to upgrade to the next version where injection is removed entirely. With this, `Inject*` functions on controllers, predicates, actuators, delegates, and friends are not called anymore. When upgrading the `gardener/gardener` dependency to this version, all injection implementations need to be removed. As a replacement, you can get the needed clients and similar from the manager during initialisation of the component.The GA-ed feature gates `HAControlPlanes` and `FullNetworkPoliciesInRuntimeCluster` have been removed.The following image is updated:
- `quay.io/prometheus/alertmanager`: `v0.24.0` -> `v0.26.0`Gardener now supports seed clusters with Kubernetes versions up to `v1.26`.gardener-apiserver: The kubelet version constraint validation is now fixed to also cover the Shoot K8s version update. Previously it was possible to update the Shoot K8s version to a new minor version when the Shoot has a worker pool with machine image version which kubeletVersionConstraint does not match the new K8s version.Now git revision and commit ids are properly propagated through build variables. These are showed in the fluent-bit plugin logs during start.A new field `.spec.virtualCluster.dns.domains` was added to the `Garden` API. This field allows to expose the `kube-apiserver` of the virtual cluster via multiple domains. Earlier, the API only accepted one domain name via `.spec.virtualCluster.dns.domain`.
⚠️ With this change `.spec.virtualCluster.dns.domain` is deprecated and will be removed in the next release. Please update your `Garden` resource to the new `.spec.virtualCluster.dns.domains` field by removing the existing domain configuration from `dns.domain` and add it as the first entry of `dns.domains`.The flags which went out-of-support in MCM v0.49.0 have been cleaned up from MCM deployment yaml.A new controller in `gardenlet` for periodically backing up the `ShootState` for `Shoot`s has been introduced. This controller is only activated when `gardenlet` is responsible for an unmanaged `Seed` (i.e., one not backed by a `ManagedSeed` object). By default, backups are taken roughly each `6h`.Change `log` mount path of `node-problem-detector` from `/var/log` to `/var/log/journal`.A new alpha feature gate `DisableScalingClassesForShoots` has been introduced on `gardenlet`. If turned on, initial resource requests for `kube-apiserver`s of shoot clusters running on seed clusters which enable the `HVPA` feature gate are assigned statically and no longer by a scaling class determined by maximum node count. This helps to reduce resource waste for clusters with little usage.Print build version and go runtime info.An issue causing panic in the health check for extension, when the health check result is empty, is fixed.A bug has been fixed which was causing the garbage collector in `gardener-resource-manager` to wrongfully collect `Secret`s related to `ManagedResource`s when the source and the target cluster are equal.When the Kubernetes control plane version is at least `v1.28`, it is now possible to set the worker pool Kubernetes version to be at most three versions behind the control plane version. Earlier, only a skew of at most two versions was allowed. Find more details [here](https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#changes-to-supported-skew-between-control-plane-and-node-versions).`gardener-operator` now takes over management of `fluent-operator` and `vali`.Added a LeaderElectionID to the controller options, allowing to run multiple instances of HVPA with leader election when `--leader-elect=true` is passed as commandline argThe `github.com/golang/mock/gomock` dependency is replaced by `go.uber.org/mock`.Now the vali ingress definition points to the shoot logging service.Improves client recreate during cluster reconcile.Introduced a new field called `machineDeploymentsLastUpdateTime` in the `Worker` status to keep track of the time when the status of the Worker resource was last updated with the latest machine deployments.⚠️ The deprecated field `.spec.settings.ownerChecks` has been removed from the Seed API. Please check your `Seed`s and remove any usage before upgrading to this Gardener version.The `{github.com/gardener/gardener/pkg/apis/core/helper,github.com/gardener/gardener/pkg/apis/core/v1beta1/helper}.SeedSettingOwnerChecksEnabled` will now return `false` if the corresponding Seed setting is `nil`. Previously, the func was returning `true` when the Seed setting is `nil`.`leader-election-resource-lock` flag is dropped and the leader-election resource-lock is hard coded to leases.It is now possible to reference `Secret`s containing kubeconfigs for admission plugins in `Shoot`s. The referenced `Secret` must be referenced in`.spec.resources` as well as in `.spec.kubernetes.kubeAPIServer.admissionPlugins[].kubeconfigSecretName`.Added pod security enforce level `baseline` label to Istio-related namespaces. The `garden` and shoot namespaces have the `privileged` level. For extension namespaces, the new `security.gardener.cloud/pod-security-standard-enforce` annotation on `ControllerRegistration` resources specifies the level. When set, the `extension` namespace is created with `pod-security.kubernetes.io/enforce` label set to `security.gardener.cloud/pod-security-standard-enforce`'s value.An issue causing `state-metrics-seed` status to show down falsely has been fixed.`gardener-operator` now refuses to start if operators attempt to downgrade or skip minor Gardener versions. Please see [this document](https://github.com/gardener/gardener/blob/master/docs/deployment/version_skew_policy.md) for more information.The `FullNetworkPoliciesInRuntimeCluster` feature gate has been promoted to beta and is now turned on by default. Before deploying this Gardener version, make sure that all your registered extensions support this feature gate.A bug has been fixed in the `garden/fluent-bit` that caused a failure in creating `networkpolicies` for scraping metrics.A bug has been fixed which could cause `kube-proxy`s from being missing after a `Shoot` has been woken up from hibernation.
gardener-robot
Release Notes: