Allow setting options per worker group

[himanshu-kun]

/kind discussion /priority 3

What would you like to be added:

Autoscaler now supports configuring certain flags per worker group. Some example flags are:

• ScaleDownUtilizationThreshold
• ScaleDownGpuUtilizationThreshold
• ScaleDownUnneededTime
• ScaleDownUnreadyTime
• MaxNodeProvisionTime

This helps in more fine-grained control over a cluster and how autoscaler works for that cluster. https://github.com/kubernetes/autoscaler/issues/3583 -> upstream issue which introduced this https://github.com/kubernetes/autoscaler/pull/3789 -> PR with first implementations

With time, more and more options are being made configurable on worker group level , which makes it essential for gardener to also allow configuring them by our end-users.

Some issue which can be solved by this are: https://github.com/gardener/autoscaler/issues/227 (by reducing scale-down-unneeded-time for the required nodegroup also)

The change should require changes in shoot API also , some related discussions are already ongoing https://github.com/gardener/gardener/issues/8142

---

Changes/PRs that need to be filed to close this story

• [x] Autoscaler: Read annotations from MCD and add them to NodeGroupAutoscalingOptions > https://github.com/gardener/autoscaler/pull/257
• [x] Gardener: Update the shoot API to expose these values to be set per node group > https://github.com/gardener/gardener/pull/9245
• [x] g/Extensions: Deploy MCDs with these node group specific values as annotations
  • [x] AWS https://github.com/gardener/gardener-extension-provider-aws/pull/901
  • [x] GCP https://github.com/gardener/gardener-extension-provider-gcp/pull/733
  • [x] Azure https://github.com/gardener/gardener-extension-provider-azure/pull/831
  • [x] Openstack https://github.com/gardener/gardener-extension-provider-openstack/pull/752
  • [x] Alicloud https://github.com/gardener/gardener-extension-provider-alicloud/pull/715
  • [x] Equinix Metal > PR not raised as vendored g/g version is 1.81. Required g/g version is at lease 1.91 https://github.com/gardener/gardener-extension-provider-equinix-metal/issues/308

Why is this needed: For better configurability of gardener autoscaler

[aaronfern]

A few options were debated, and we finally decided to go with an approach that is similar to the one used by amazon ASG

The following is a proposal of the solution

g/g changes:

• Add spec.provider.workers.clusterAutoscaler to the shoot API that will expose these 5 fields to be set per node group

  example

  ``` spec: provider: workers: - name: test-name clusterAutoscaler: scaleDownUtilizationThreshold: 0.5 scaleDownGpuUtilizationThreshold: 0.5 scaleDownUnneededTime: 1m scaleDownUnreadyTime: 1m maxNodeProvisionTime: 1m ```

g/extension changes:

• Update code which creates and deploys machineDeployments to add these node group specific values as annotations on the machineDeployment. If node group specific values are not provided then shoot defaults are to be added as annotations.

autoscaler changes:

• Update the machineDeployment.GetOptions() method in mcm_cloud_provider.go to read the annotations on machineDeployments and add these node group specific options to the NodeGroupAutoscalingOptions struct

The other approached we discussed were the following:

• Overload the --nodes flag to add these node group specific values. This then involves us updating the validation for this flag, and we would then be at a disadvantage should we go down a path of using node-group-auto-discovery
• Adding a new flag (for eg --node-group-options) that contains node group specific options , but this option has similar problems as the above, plus its making change to the core autoscaler which we should avoid.

/cc @timuthy for comments on the g/g changes

[rishabh-11]

/close as all PRs are merged