Increment min-servers does not scale up the cloudProviderTarget (Hetzner provider)

[mickare]

Which component are you using?:

cluster-autoscaler with Hetzner provider

What version of the component are you using?:

Tested two latest commits:

• master (5e85a7d)
• Latest PR to Hetzner provider (23088ae)

Component version:

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"archive", BuildDate:"2022-08-08T17:09:48Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?:

Hetzner Cloud running a single K3S master

What did you expect to happen?:

The nodes in pool pool1 scale up to the minimum size.

What happened instead?:

• The pool does not scale.
• The autoscaler logs do not show any attempt to scale up to the target count.
• The cluster-autoscaler-status shows cloudProviderTarget=1 although minSize=3.

How to reproduce it (as minimally and precisely as possible):

1. Create cluster with example --nodes=1:10:CPX11:FSN1:pool1
2. Wait for pool to autoscale to one node "pool1-..."
3. Update deployment and increment the pool min-servers to 3:

   • $ kubectl edit deployments -n kube-system cluster-autoscaler

   • --nodes=3:10:CPX11:FSN1:pool1
4. Hope that the autoscaler is working:

   • $ kubectl logs -n kube-system cluster-autoscaler-...

Anything else we need to know?:

Related issue: Scale up does not happen when current node count is smaller than min size #4942

$ kubectl describe configmaps -n kube-system cluster-autoscaler-status                                                                 [130]
Name:         cluster-autoscaler-status
Namespace:    kube-system
Labels:       <none>
Annotations:  cluster-autoscaler.kubernetes.io/last-updated: 2022-09-02 21:53:15.83258693 +0000 UTC

Data
====
status:
----
Cluster-autoscaler status at 2022-09-02 21:53:15.83258693 +0000 UTC:
Cluster-wide:
  Health:      Healthy (ready=2 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=2 longUnregistered=0)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
  ScaleUp:     NoActivity (ready=2 registered=2)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
  ScaleDown:   NoCandidates (candidates=0)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198

NodeGroups:
  Name:        draining-node-pool
  Health:      Healthy (ready=0 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=0 longUnregistered=0 cloudProviderTarget=0 (minSize=0, maxSize=0))
               LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
               LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
  ScaleUp:     NoActivity (ready=0 cloudProviderTarget=0)
               LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
               LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
  ScaleDown:   NoCandidates (candidates=0)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198

  Name:        pool1
  Health:      Healthy (ready=1 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=1 longUnregistered=0 cloudProviderTarget=1 (minSize=3, maxSize=10))
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
  ScaleUp:     NoActivity (ready=1 cloudProviderTarget=1)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
  ScaleDown:   NoCandidates (candidates=0)
               LastProbeTime:      2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
               LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198

BinaryData
====

Events:  <none>

$ kubectl logs -n kube-system pods/cluster-autoscaler-7...
I0902 21:31:40.004115       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0902 21:31:57.431369       1 leaderelection.go:258] successfully acquired lease kube-system/cluster-autoscaler
W0902 21:31:58.484305       1 hetzner_servers_cache.go:94] Fetching servers from Hetzner API
I0902 21:31:58.657834       1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
I0902 21:31:58.657886       1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 28.573µs
I0902 21:32:08.658179       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:08.658216       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
W0902 21:32:08.658579       1 hetzner_server_type_cache.go:93] Fetching server types from Hetzner API
W0902 21:32:08.701442       1 clusterstate.go:423] AcceptableRanges have not been populated yet. Skip checking
W0902 21:32:08.701472       1 clusterstate.go:423] AcceptableRanges have not been populated yet. Skip checking
I0902 21:32:18.714726       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:18.714775       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:28.729157       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:28.729187       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:38.742996       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:38.743030       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:48.756958       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:48.756994       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
W0902 21:32:58.770358       1 hetzner_servers_cache.go:94] Fetching servers from Hetzner API
I0902 21:32:58.907363       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:58.907404       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:33:08.923410       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:33:08.923451       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:33:18.937652       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:33:18.937695       1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
...

[timowevel1]

I can confirm it doesnt work at all