What k8s version are you using (kubectl version)?:
kubectl version Output
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"archive", BuildDate:"2022-08-08T17:09:48Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
What environment is this in?:
Hetzner Cloud running a single K3S master
What did you expect to happen?:
The nodes in pool pool1 scale up to the minimum size.
What happened instead?:
The pool does not scale.
The autoscaler logs do not show any attempt to scale up to the target count.
The cluster-autoscaler-status shows cloudProviderTarget=1 although minSize=3.
How to reproduce it (as minimally and precisely as possible):
Create cluster with example--nodes=1:10:CPX11:FSN1:pool1
Wait for pool to autoscale to one node "pool1-..."
Update deployment and increment the pool min-servers to 3:
$ kubectl describe configmaps -n kube-system cluster-autoscaler-status [130]
Name: cluster-autoscaler-status
Namespace: kube-system
Labels: <none>
Annotations: cluster-autoscaler.kubernetes.io/last-updated: 2022-09-02 21:53:15.83258693 +0000 UTC
Data
====
status:
----
Cluster-autoscaler status at 2022-09-02 21:53:15.83258693 +0000 UTC:
Cluster-wide:
Health: Healthy (ready=2 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=2 longUnregistered=0)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
ScaleUp: NoActivity (ready=2 registered=2)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
NodeGroups:
Name: draining-node-pool
Health: Healthy (ready=0 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=0 longUnregistered=0 cloudProviderTarget=0 (minSize=0, maxSize=0))
LastProbeTime: 0001-01-01 00:00:00 +0000 UTC
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
ScaleUp: NoActivity (ready=0 cloudProviderTarget=0)
LastProbeTime: 0001-01-01 00:00:00 +0000 UTC
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
Name: pool1
Health: Healthy (ready=1 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=1 longUnregistered=0 cloudProviderTarget=1 (minSize=3, maxSize=10))
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
ScaleUp: NoActivity (ready=1 cloudProviderTarget=1)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2022-09-02 21:53:15.831855274 +0000 UTC m=+1296.583516420
LastTransitionTime: 2022-09-02 21:32:08.658045042 +0000 UTC m=+29.409706198
BinaryData
====
Events: <none>
$ kubectl logs -n kube-system pods/cluster-autoscaler-7...
I0902 21:31:40.004115 1 leaderelection.go:248] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0902 21:31:57.431369 1 leaderelection.go:258] successfully acquired lease kube-system/cluster-autoscaler
W0902 21:31:58.484305 1 hetzner_servers_cache.go:94] Fetching servers from Hetzner API
I0902 21:31:58.657834 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
I0902 21:31:58.657886 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 28.573µs
I0902 21:32:08.658179 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:08.658216 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
W0902 21:32:08.658579 1 hetzner_server_type_cache.go:93] Fetching server types from Hetzner API
W0902 21:32:08.701442 1 clusterstate.go:423] AcceptableRanges have not been populated yet. Skip checking
W0902 21:32:08.701472 1 clusterstate.go:423] AcceptableRanges have not been populated yet. Skip checking
I0902 21:32:18.714726 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:18.714775 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:28.729157 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:28.729187 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:38.742996 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:38.743030 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:32:48.756958 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:48.756994 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
W0902 21:32:58.770358 1 hetzner_servers_cache.go:94] Fetching servers from Hetzner API
I0902 21:32:58.907363 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:32:58.907404 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:33:08.923410 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:33:08.923451 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
I0902 21:33:18.937652 1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0902 21:33:18.937695 1 hetzner_node_group.go:437] Set node group pool1 size from 1 to 1, expected delta 0
...
Which component are you using?:
cluster-autoscaler with Hetzner provider
What version of the component are you using?:
Tested two latest commits:
Component version:
What k8s version are you using (
kubectl version
)?:kubectl version
OutputWhat environment is this in?:
Hetzner Cloud running a single K3S master
What did you expect to happen?:
The nodes in pool
pool1
scale up to the minimum size.What happened instead?:
cluster-autoscaler-status
showscloudProviderTarget=1
althoughminSize=3
.How to reproduce it (as minimally and precisely as possible):
--nodes=1:10:CPX11:FSN1:pool1
--nodes=3:10:CPX11:FSN1:pool1
Anything else we need to know?:
Related issue: Scale up does not happen when current node count is smaller than min size #4942