Adjust shoot spec for cluster created in region me-central2 to use assured workloads by setting tolerations-field

[tobiscr]

Description

Caused by legal requirements is an adjustment for SKR clusters in SA required:

It has to be ensured, that all SKR clusters in this region are using assured workloads. This can be achieved by configuring the tolerations-field:

spec:
  tolerations:
    - key: ksa-assured-workload

We agreed with colleagues from KEB to set this field for each cluster which should be scheduled for region me-central2.

AC:

• [ ] Provisioner is setting the tolerations field is set in Shoot-spec for each cluster whcih gets scheduled in region me-central2
• [ ] Should be done for cluster creation ~and update~

Potential risk: The Gardener operator has to accept the toleration. This needs initially be configured by the Gardener team for this Gardener project.

Reasons

Being compliant with KSA restrictions for SKR clusters in SA.

Attachments

[tobiscr]

@varbanv : we would apply this parameter also to existing cluster (would require a recreation by Gardener) - can we implement the change or do we have to ensure that existing clusters are not touched?

[varbanv]

@tobiscr we have to ask these users to re-create the clusters, we can't move them. What happens if you apply the change? Would Gardener completely destroy those clusters?

[tobiscr]

@varbanv : But afaik there is an option to apply the change only for new created clusters (@koala7659 - please correct me if this is wrong). Means: new clusters will get the fix, but old clusters won't get the fix. This implies, that existing clusters have to be re-created by customers to receive the fix.

Alterantively, we apply the fix for new and existing clusters. This would cause an update of existing Shoot-Specs and Gardener would re-create these clusters.

[varbanv]

@tobiscr if there is a "only for new clusters" option then let's do that. The old ones can't be migrated so they'll have to be re-created anyway.

[tobiscr]

Ok, so we will apply the fix ONLY FOR NEW CREATED CLUSTERS and not for existing clusters.

[mvshao]

@tobiscr the potential risk criteria are met

Potential risk:
The Gardener operator has to accept the toleration. This needs initially to be configured by the Gardener team for this Gardener project.

The Gardener team needs to accept this toleration. Error from Gardener client during provision call

error while validating tolerations against allowlist: [spec.tolerations[0]: Forbidden: only the following tolerations are allowed: [seed.gardener.cloud/alpha-ha]]

I will contact the Gardener team to add support for this toleration,

[mvshao]

The changes in Provisioner are now on the Dev environment. We need to wait for propagation to Stage and Prod.