kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
7.98k stars 3.94k forks source link

Linode cluster-autoscaler not working as expected #4192

Closed Yapcheekian closed 3 years ago

Yapcheekian commented 3 years ago

Which component are you using?: cluster-autoscaler

What version of the component are you using?: v1.21.0

What k8s version are you using (kubectl version)?:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider: Linode

What did you expect to happen?: I have pods that are in pending state.

NAME                           READY   STATUS    RESTARTS   AGE
internalsys-69b9d9c495-2qr7s   0/1     Pending   0          4m15s
internalsys-69b9d9c495-2rgl6   1/1     Running   0          4m57s
internalsys-69b9d9c495-5smnt   0/1     Pending   0          4m15s
internalsys-69b9d9c495-6kx4v   0/1     Pending   0          4m15s
internalsys-69b9d9c495-6v7q6   1/1     Running   0          4m57s
internalsys-69b9d9c495-77v6k   0/1     Pending   0          4m15s
internalsys-69b9d9c495-8k24w   1/1     Running   0          4m57s
internalsys-69b9d9c495-9cgtd   0/1     Pending   0          4m15s
internalsys-69b9d9c495-c2pfs   0/1     Pending   0          4m15s
internalsys-69b9d9c495-d86xx   1/1     Running   0          4m57s

this is the log of cluster-autoscaler:

I0713 09:14:43.751284       1 linode_manager.go:65] The LKE pool 47319 has more than one node (current nodes in pool: 2), will exclude it from the node groups
I0713 09:14:43.751308       1 linode_manager.go:84] LKE node group after refresh:
I0713 09:14:43.757041       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-g8m8b is unschedulable
I0713 09:14:43.757067       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-vnpvz is unschedulable
I0713 09:14:43.757070       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-gpqtr is unschedulable
I0713 09:14:43.757073       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-kd45v is unschedulable
I0713 09:14:43.757076       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-z29ks is unschedulable
I0713 09:14:43.757078       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-5smnt is unschedulable
I0713 09:14:43.757081       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-2qr7s is unschedulable
I0713 09:14:43.757083       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-kbh6f is unschedulable
I0713 09:14:43.757086       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-fdbw8 is unschedulable
I0713 09:14:43.757089       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-6kx4v is unschedulable
I0713 09:14:43.757091       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-zpwb7 is unschedulable
I0713 09:14:43.757093       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-vxjwc is unschedulable
I0713 09:14:43.757096       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-9cgtd is unschedulable
I0713 09:14:43.757099       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-dqns5 is unschedulable
I0713 09:14:43.757101       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-j6x9b is unschedulable
I0713 09:14:43.757104       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-77v6k is unschedulable
I0713 09:14:43.757106       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-jvgpv is unschedulable
I0713 09:14:43.757109       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-c2pfs is unschedulable
I0713 09:14:43.757111       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-stlk9 is unschedulable
I0713 09:14:43.757113       1 klogx.go:86] Pod staging/internalsys-69b9d9c495-rkdr4 is unschedulable
I0713 09:14:43.761945       1 scale_up.go:453] No expansion options

I'm expecting LKE node pools to be added to my node groups, but this doesn't happen.

Here is my cloud-config:

[global]
    linode-token=xxx
    lke-cluster-id=30700
    defaut-min-size-per-linode-type=1
    defaut-max-size-per-linode-type=10
    [nodegroup "g6-standard-2"]
    min-size=2
    max-size=5
displague commented 3 years ago

@Yapcheekian, I'm curious what happened here if you recall why you opened this and closed it.

Yapcheekian commented 3 years ago

I missed the part in the doc saying that I need to create a node pool to hold only a single linode for it to become a valid LKE scaling node pool.

At every scan interval the cluster autoscaler reviews every LKE Node Pool and if:

- it is not among the ones to be excluded as defined in the configuration file;
- it holds a single Linode;

After trying this, everything works fine.