gardener-robot-ci-3
commented
1 year ago e/shoot_cleanup.md)) for more information.
```breaking operator github.com/gardener/etcd-druid #681 @aaronfern
Etcd druid will now not support `policy/v1beta1` for `PodDisruptionBudget`s and will only use `policy/v1` for `PodDisruptionBudget`sThe `NetworkPolicy` reconciler is only added to `gardener-operator` if the `.spec.runtimeCluster.networking.{pods,services}` fields of the `Garden` are set.Added a new metric that will allow to get the number of stale (due to unhealthiness) machines that are getting terminated`pkg/resourcemanager/controller/garbagecollector/references.InjectAnnotations` now also handles `pods.spec.imagePullSecrets`. `operator` now deletes `ManagedResources` deployed to the virtual-garden before deleting `virtual-garden-kube-apiserver`.A bug causing the gardenlet to panic when a ETCD encryption key rotation operation is triggered for a hibernated Shoot is now fixed. Now, triggering ETCD encryption key rotation or ServiceAccount signing key rotation is forbidden when the Shoot is in waking up phase.Backupbucket/backupentry controllers: watch secret metadata onlyUpdate to golang v1.21Annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns` is deprecated and will be removed in future releases. Use field `.spec.systemComponents.nodeLocalDNS.enabled` in `Shoot` instead. Switching on node-local-dns via shoot specification will roll the nodes even if node-local-dns was enabled beforehand via annotation.Provider extensions should be adapted such that they no longer perform health checks specific to the `machine-controller-manager` deployment or the machines/nodes. In the future, `gardenlet` will take over performing these checks. Please see https://github.com/gardener/gardener/pull/8019 for an example how `provider-local` was adapted and replicate it for your provider extensions.A bug has been fixed for Istio-Ingress Gateways for seeds that use `ExposureClassHandler`s. Earlier, annotations in `seed.spec.settings.loadBalancerServices` caused an override of the ones specified in `gardenletConfiguration.exposureClassHandler[].loadBalancerService` for zonal Istios. Now, annotations in `gardenletConfiguration.exposureClassHandler[].loadBalancerService` are given priority, like it was already the case of the global Istio.`Secret`s/`ConfigMap`s referenced in `.spec.resources` of `Shoot`s are now protected with a finalizer to ensure they do not disappear from the system as long as they are still referenced somewhere.`nginx-ingress-controller` image is updated to `v1.9.0`.The `extensions/pkg/controller.Use{TokenRequestor,ServiceAccountTokenVolumeProjection}` functions have been removed since they always return `true`.When the Kubernetes control plane version is at least `v1.28`, it is now possible to set the worker pool Kubernetes version to be at most three versions behind the control plane version. Earlier, only a skew of at most two versions was allowed. Find more details [here](https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#changes-to-supported-skew-between-control-plane-and-node-versions).Update gardener/gardener to 1.77.1.HVPA supports k8s versions >= 1.25 by switching to `k8s.io/autoscaling/v2` when necessary for all API calls.Plutono is now updated to v7.5.22The `spec.secretBindingName`, `.spec.networking`, `.spec.networking.type`, `spec.maintenance.autoUpdate.machineImageVersion` fields in the Shoot API are now made optional to prepare for the introduction of workerless Shoots feature. Please see https://github.com/gardener/gardener/issues/7635 for more details.As of Kubernetes `v1.27`, Gardener enforces a `worker.maximum` configuration for system component worker pools. The value must be greater or equal to the number of zones configured for this pool. This ensures, that the pool has the minimum required nodes to schedule system component across nodes.Update golang 1.19.5 -> 1.20.4Update Prometheus job `tunnel-probe-apiserver-proxy` to fix for HA VPN modeDruid now exposes metrics related to snapshot compaction, on default port 8080. Please expose the desired metrics port via the etcd-druid service to allow metrics to be scraped by a Prometheus instance.Deprecated annotation `alpha.featuregates.shoot.gardener.cloud/node-local-dns` is removed. Use field `.spec.systemComponents.nodeLocalDNS.enabled` in `Shoot` instead. Switching on node-local-dns via shoot specification will roll the nodes even if node-local-dns was enabled beforehand via annotation.Use admission v1 instead of v1beta1 for apiserver-proxy webhook.The `--node-monitor-grace-period` flag of `kube-controller-manager` is now defaulted to `40s` for Shoot clusters using Kubernetes version `1.27` and higher.The `.{source,target}ClientConnection.namespace` field has been renamed to `namespaces` and now takes a list of namespaces. The `.targetClientConnection.disableCachedClient` field has been removed.A configuration issue that resulted in a relatively slow startup and termination of the vali pods is fixed.`gardenlet` is taking over management of the `CustomResourceDefinition`s for the `machine.sapcloud.io/v1alpha1` API group, hence extensions do no longer need to take care. Consequently, the `extensions/pkg/controller/worker.Options` struct as well as the `extensions/pkg/controller/worker.ApplyMachineResources{ForConfig}` functions are deprecated and will be removed in a future release. A guideline for developers regarding [`TODO` statements](https://github.com/gardener/gardener/blob/docs/master/development/process.md#todo-statements) has been introduced.extension library: State update for a Worker object can be now skipped by annotating it with `worker.gardener.cloud/skip-state-update=true`.Fixed an issue that would cause the `gardenlet` to run into `CrashLoopBackoff` when following the docs/development/getting_started_locally.md#remote-local-setup guide.Etcd-druid will now deploy distroless `etcd-wrapper` and `etcd-backup-restore` images. Please refer to [etcd-wrapper](https://github.com/gardener/etcd-wrapper) for more information.Add CVE categorization for etcd-backup-restore.The static token kubeconfig can no longer be enabled for Shoot clusters using Kubernetes version `1.27` and higher.The `core/v1alpha1` API version is dropped. Make sure that you don't use the `core/v1alpha1` API version in your machinery.A bug was fixed which was causing existing `Bastion` resources on the garden cluster to not be deleted when `SSHAccess` is disabled on a Shoot cluster.All components in the gardener logging stack are now updated to the following respective versions. Fluent-bit to 2.1.4, Fluent-operator to 2.3.0 and logging to 0.55.3Updated kubernetes dependencies from `1.25.0` to `1.26.2`The two additional labels `worker.gardener.cloud/image-name` and `worker.gardener.cloud/image-version` that were previously introduced and attached to worker nodes are removed again to fix a regression that causes the `kubelet` to restart on nodes that are due to be upgraded to a new OS but not rolled yet which causes their `Pod`s to become temporarily unready.The project vendors the latest released gardener version - v1.73.0If `gardenlet` is responsible for a managed `Seed`, it will delete all `ShootState` resources for its `Shoot`s that are not currently in migration. See also [GEP-22](https://github.com/gardener/gardener/blob/master/docs/proposals/22-improved-usage-of-shootstate-api.md) for further details about the motivation.The admission controllers of common provider extensions are automatically installed in the local extensions development setupIt is now easier to annotate `Service`s related to extensions serving webhook handlers that must be reached by `kube-apiserver`s running in separate namespaces such that the respective network traffic gets allowed. Please refer to [this guide](https://github.com/gardener/gardener/blob/master/docs/usage/network_policies.md#webhook-servers) for all information. Extensions serving shoot webhook should make use of this new approach - the old functionality deploying dedicated `NetworkPolicy`s is deprecated and will be removed in the future.New metrics introduced:
- api_request_duration_seconds -> tracks time taken for successful invocation of provider APIs. This metric can be filtered by provider and service.
- driver_request_duration_seconds -> tracks total time taken to successfully complete driver method invocation. This metric can be filtered by provider and operation.
- driver_requests_failed_total -> records total number of failed driver API requests. This metric can be filtered by provider, operations and error_code.A new controller in `gardenlet` for periodically backing up the `ShootState` for `Shoot`s has been introduced. This controller is only activated when `gardenlet` is responsible for an unmanaged `Seed` (i.e., one not backed by a `ManagedSeed` object). By default, backups are taken roughly each `6h`.`nginx-ingress-controller-seed` image is updated to `v1.8.0` for `1.24.x+` seeds.The reconciliation time limit for the controller resource reconciliation, e.g. for `ManagedResource`, has been increased from `1m` to `3m`.Now git revision and commit ids are properly propagated through build variables. These are showed in the fluent-bit plugin logs during start.Bump alpine base version for Docker build to `3.18.2`.Added `errorCode` field in the `LastOperation` struct. This should be implemented only for the `CreateMachine` call in the `triggerCreationFlow`. This field will be utilized by Cluster autoscaler to do early backoff Fix verification.The `DisablingScalingClassesForShoots` feature gate has been promoted to beta. Fix an issue, where DNS lookups for non-existing pods of a StatefulSet yielded one of the existing pods even when it should not have. A bug is fixed in the Prometheus alert definitions that caused false positive KubePodNotReadyControlPlane alerts related to the etcd compaction job.The `gardener-operator` does now also manage `kube-state-metrics`.Backup-restore waits for its etcd to be ready before attempting to update peerUrlAdd CVE categorization for etcd-backup-restore.It is possible now to trigger a seed reconciliation by annotating the Seed with `gardener.cloud/operation=reconcile`.A new alpha feature gate named `MachineControllerManagerDeployment` has been introduced in `gardenlet`. Only enable it when all registered provider extensions in your landscape support this feature.The kind cluster used in local setup does now use the new way in containerd to configure registry mirrors.gardener-apiserver now exposes a new `core.gardener.cloud/v1beta1.InternalSecret` API, see the [documentation](https://github.com/gardener/gardener/blob/master/docs/concepts/apiserver.md#internalsecrets) for more information.Makefile has been updated to use `Skaffold` for deploying `etcd-druid` with the `make deploy` target, simplifying the deployment process and eliminating the need to push the image to the container registry for each local development testing.Add failure tolerance option to the `CreateShoot` test.Eliminated `RoleBinding` helm charts and converted into Golang component with added unit tests.The `fluent-bit-vali-plugin` now supports fluent-bit v2.1.0 and above.A new optional constraint `CRDsWithProblematicConversionWebhooks` is introduced in the `Shoot` status. This constraint indicates that there is at least one CRD in the cluster which has multiple stored versions and a conversion webhook configured, which could break the reconciliation flow of a `Shoot` in some cases.A bug has been fixed that prevented `ControllerInstallation`s from getting deleted when the backing `ControllerRegistration` with `.spec.deployment.policy={Always,AlwaysExceptNoShoots}` was deleted.update client-go version and exclude the old one in go.modA new field `.spec.virtualCluster.dns.domains` was added to the `Garden` API. This field allows to expose the `kube-apiserver` of the virtual cluster via multiple domains. Earlier, the API only accepted one domain name via `.spec.virtualCluster.dns.domain`.
⚠️ With this change `.spec.virtualCluster.dns.domain` is deprecated and will be removed in the next release. Please update your `Garden` resource to the new `.spec.virtualCluster.dns.domains` field by removing the existing domain configuration from `dns.domain` and add it as the first entry of `dns.domains`.The `shootstate-extensions` and `shootstate-secret` controllers have been dropped. The `gardenlet`'s component config file should be updated to no longer specify related configuration (`.controllers.{shootSecret,shootStateSync}`).Introduced a new field called `machineDeploymentsLastUpdateTime` in the `Worker` status to keep track of the time when the status of the Worker resource was last updated with the latest machine deployments.Change `log` mount path of `node-problem-detector` from `/var/log` to `/var/log/journal`.Force drain and delete volume attachments for nodes un-healthy due to `ReadOnlyFileSystem` and `NotReady` for too long`gardener-operator` is now managing the `gardener-resource-manager` instance as part of the virtual garden cluster control plane. It provides a `TokenRequest` API-based kubeconfig for `gardener-operator` to access the virtual garden cluster. The static token kubeconfig is now unconditionally disabled.Stability of the ssh tunnel in the local extension setup should improve due to better failure handling.The `core/v1alpha1` API version is dropped. Before upgrading to this version, make sure that there are no resources in the etcd stored in the `core/v1alpha1` API version. Otherwise, the gardener-apiserver@v1.72.0 will fail to start.Operators can now view and manage dashboards for compaction jobs running in shoot control plane.Extensions vendoring this `gardener/gardener` version need to provide RBAC privileges for `PATCH apps/depoyments/scale`.The `gardener-scheduler` now populates scheduling failure reasons to the `Shoot`'s `.status.lastOperation.description` field.A regression was fixed that prevented deletions for shoot clusters which were created with a wrong configuration (e.g. with an unavailable domain name).The regression is now fixed and the control plane logs shall be visible in the Plutono dashboards.A new feature gate named `ContainerdRegistryHostsDir` is introduced to gardenlet. When enabled, the `/etc/containerd/certs.d` directory is created on the Node and containerd is configured to look up for registries/mirrors configuration in this directory (if there is any configuration applied). In future, the [registry-cache extension](https://github.com/gardener/gardener-extension-registry-cache/) will add such registries/mirrors configuration under this directory (via OperatingSystemConfig mutation).Shoot fields `.spec.dns.providers[].domains` and `.spec.dns.providers[].zones` are now deprecated and expected to be removed in version `v1.87`. Please plan ahead to drop using those fields in extensions.The `Deploying Shoot namespace in Seed` step was slightly improved. Earlier it failed at some occasions when it tried to read zone information for volumes that have not been created yet. This was a transient error that dissolved in subsequent reconcile runs.Improves client recreate during cluster reconcile.A bug is fixed that prevented scraping the metrics of etcd in the shoot control plane.The `gardenlet` and `gardener-operator` Helm charts allow to define toleration seconds for `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable`. This configuration considered for their own Deployment as well as the Gardenlet's or Operator's config. The values are set to `60s` by default.Shoot addon `nginx-ingress-controller` image is updated to `v1.8.0` for Kubernetes `v1.24+` clusters, to `v1.6.4` for Kubernetes `v1.23` clusters, and to `v1.4.0` for Kubernetes `v1.22` clusters.`extensions.gardener.cloud/v1alpha1.ControlPlane` is now deployed after `kube-apiserver` in the Shoot reconciliation flow.Following dependencies are updated:
Go - 1.20.3
client-go - v0.26.2
controller-runtime - v0.14.5
gomega - v1.27.1
zap - v1.24.0
gardener/gardener v1.69.0
k8s (api and apimachinery) - v0.26.2The unused `github.com/gardener/gardener/pkg/controllerutils/predicate.IsBeingMigratedPredicate`, `github.com/gardener/gardener/pkg/controllerutils/predicate.IsObjectBeingMigrated` and `github.com/gardener/gardener/pkg/utils/gardener.IsObjectBeingMigrated` funcs are now removed.`gardenlet` no longer reports the `Bootstrapped` condition on `Seed`s. Instead, it now reports the progress in `.status.lastOperation`, similar to how it's done for `Shoot`s.Upgrade to go 1.20.3Refactored `statefulset`, `service`, `poddisruptionbudget`, `lease`, and `configmap` components to use default labels and owner references from `etcd`.The `Shoot` maintenance controller now updates the CRI of worker pools from `docker` to `containerd` when force-upgrading from Kubernetes `v1.22` to `v1.23`.`AllMembersReady` condition has now been fixed to eventually show the correct overall readiness of an etcd cluster.Gardener now allows to omit or to only partially define Kubernetes versions in `Shoot`s. The version will automatically be defaulted to the latest minor and/or patch version found in the linked `CloudProfile`.Bump builder image golang from `1.20.4` to `1.20.6` The `alpha.kube-apiserver.scaling.shoot.gardener.cloud/class` annotation on `Shoot`s has no effect anymore and should be removed.Before upgrading to this gardener version, operators should configure `gardener-apiserver` to encrypt the `internalsecrets.core.gardener.cloud` resource in etcd.The `check-apidiff` check was changed to only report incompatible and critical changes which need inspection from the developer's side.`maintenance-controller` now disables `PodSecurityPolicy` admission controller when forcefully upgrading the Kubernetes version of a `Shoot` to `v1.25`. It also ensures maximum workers of each for group is greater or equal to its number of zone for forceful upgrades to `v1.27`.Using internal API versions in `providerConfig` fields is no longer permitted (deprecated since more than `2y`). Ensure that you always use a versioned API.Upgrade gardener/gardener from `1.65.0` to `1.76.0``custodian-sync-period` value is set to `15s` in the Helm chart for etcd-druid.Applying Gardener resources server-side has caused the `the server is currently unable to handle the request` error which is now fixed.A new field `errorCodeCheckFunc` is introduced in the generic `Worker` actuator. This should be set to parse the Gardener error codes from the error returned in `Worker` reconciliation.Bump alpine base version for Docker build to `3.18.2`. An edge case where outdated DesiredReplicas annotation blocked a rolling update is fixed.Any resource with a kind other than `ConfigMap` or `Secret` in `.spec.resources` in `Shoot`s is now forcefully removed. New validation has been introduced to prevent adding other resources in the future.`Shoot`s allow to optionally configure a specific scheduler via `.spec.schedulerName`. The `default-scheduler` is used in case non is configured. Please note, that `Shoot`s will remain `Pending` in case a scheduler name is configured but an adequate scheduler is not available in the landscape.Run `make ci-e2e-kind` to run the e2e tests on local machineThe flags which went out-of-support in MCM v0.49.0 have been cleaned up from MCM deployment yaml.`machinepriority.machine.sapcloud.io` annotation on machine is now reset to 3 by autoscaler if the corresponding node doesn't have `ToBeDeletedByClusterAutoscaler` taintPackage `pkg/utils/managedresources` now works with immutable secrets for managed resources under the hood. Existing secrets will be marked for garbage collection and replaced with immutable ones during the first reconciliation of the managed resource.⚠️ Gardener does no longer support garden, seed, or shoot clusters with Kubernetes versions < 1.24. Make sure to upgrade all existing clusters before upgrading to this Gardener version.It is now possible to configure `.spec.virtualCluster.gardener.gardenerAPIServer.auditWebhook` in the `Garden` API.An issue has been fixed for highly-available `Shoot`s whose `etcd` clusters didn't get ready in the `Completing` phase of a CA credentials rotation.Updated to go v1.20.5Bump `k8s.io/*` deps to v0.27.2If the `kubeletCSRApprover` controller is enabled, it is now mandatory to specify the namespace in the source cluster in which the `Machine` resources reside via `.controllers.kubeletCSRApprover.machineNamespace`.gardenlet: A regression preventing the alertmanager in the garden namespace from sending email notifications is now fixed.Prometheus scrape job configs for targets in the shoot cluster have been improved.Gardener uses an `InternalSecret` per Shoot for syncing the client CA to the project namespace in the garden cluster (named `<shoot-name>.ca-client`). The `shoots/adminkubeconfig` subresource signs short-lived client certificates by retrieving the CA from the `InternalSecret`.The following golang dependencies have been upgraded, please consult the upstream release notes and [this issue](https://github.com/gardener/gardener/issues/8382) for guidance on upgrading your golang dependencies when vendoring this gardener version:
- `k8s.io/*` to `v0.28.2`
- `sigs.k8s.io/controller-runtime` to `v0.16.2`
- `sigs.k8s.io/controller-tools` to `v0.13.0`The following images are updated:
- registry.k8s.io/metrics-server/metrics-server: v0.6.3 -> v0.6.4
- registry.k8s.io/cpa/cluster-proportional-autoscaler: v1.8.8 -> v1.8.9
- registry.k8s.io/coredns/coredns: v1.10.0 -> v1.10.1
- quay.io/prometheus/blackbox-exporter: v0.23.0 -> v0.24.0
- quay.io/prometheus/node-exporter: v1.5.0 -> v1.6.1
- ghcr.io/credativ/plutono: v7.5.22 -> v7.5.23
- ghcr.io/prometheus-operator/prometheus-config-reloader: v0.61.1 -> v0.67.1
- registry.k8s.io/dns/k8s-dns-node-cache: 1.22.20 -> 1.22.23⚠️ The deprecated field `.spec.settings.ownerChecks` has been removed from the Seed API. Please check your `Seed`s and remove any usage before upgrading to this Gardener version.The following images are updated:
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.21.5` -> `v1.21.6` (for Kubernetes `1.21`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.22.5` -> `v1.22.6` (for Kubernetes `1.22`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.23.3` -> `v1.23.4` (for Kubernetes `1.23`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.24.2` -> `v1.24.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.25.2` -> `v1.25.3` (for Kubernetes `1.24`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.26.1` -> `v1.26.2` (for Kubernetes `1.26`)A bug in the local development environment has been fixed which prevented admission of Gardener resources by extension webhooks.A bug causing the shoot provider label in the infrastructure secret to not get cleaned up is now fixed.Removed `service.beta.kubernetes.io/aws-load-balancer-type: nlb` annotation from istio-ingressgateway service template. Set this annotation in seed configuration. Note: Changing load balancer type creates a new one, old one requires manual clean-up.The shoot namespace in seeds is redeployed during the shoot migration flow to update the zones in use.The `shoots/adminkubeconfig` relies on the `ca-client` `InternalSecret` only and does not use the `ShootState` object anymore.⚠️ Gardener does no longer support garden, seed, or shoot clusters with Kubernetes versions < 1.22. Make sure to upgrade all existing clusters before upgrading to this Gardener version.A bug which was causing race conditions to occur during reconciliation of extension resources was fixed. `gardener-operator` is now managing the Gardener control plane components (`gardener-{apiserver,admission-controller,controller-manager,scheduler}`).Updated go to 1.19.9`pkg/utils/chart` does now support embedded charts. The already deprecated methods in the `ChartApplier` and `ChartRenderer` will be removed in a few releases, so extensions should adapt to embedded charts.An issue causing panic in the health check for extension, when the health check result is empty, is fixed.The deprecated feature gate `APIServerSNI` has been removed.Missing permissions were added for the Gardenlet service account for `Machine` objects. This fix is relevant if feature gate `MachineControllerManagerDeployment` is enabled in your landscape.Makefile targets have changed: Introduced gardener-setup, gardener-restore, gardener-local-mcm-up, non-gardener-setup, non-gardener-restore, non-gardener-local-mcm-up. Users can also directly use the scripts which are used by these makefile targets.
gardener-robot
Release Notes: