search

kubernetes-retired / kube-aws

[EOL] A command-line tool to declaratively manage Kubernetes clusters on AWS
Apache License 2.0
1.12k stars 295 forks source link

Autoscaller doesn't scale down after utilisation. #1302

Closed nandakishorebhandari closed 6 years ago

nandakishorebhandari commented 6 years ago

I have cluster with: 1 etcd node 1 worker node 1 controller node.

I have enabled cluster autoscaling with min 1 node and max 3 nodes and 2 nodes are rolling update. Cluster scaled up successfully, but after utilization, nodes not got deleted even with zero utilization.

Logs of cluster-autoscaller pod: I0514 08:53:03.044090 1 polling_autoscaler.go:111] Poll finished I0514 08:53:03.044110 1 static_autoscaler.go:97] Starting main loop I0514 08:53:03.121935 1 static_autoscaler.go:230] Filtering out schedulables I0514 08:53:03.122013 1 static_autoscaler.go:240] No schedulable pods I0514 08:53:03.122028 1 static_autoscaler.go:244] No unschedulable pods I0514 08:53:03.122038 1 static_autoscaler.go:280] Calculating unneeded nodes I0514 08:53:03.149494 1 utils.go:407] Skipping ip-10-0-0-231.ec2.internal - node group min size reached I0514 08:53:03.149510 1 utils.go:398] Skipping ip-10-0-0-48.ec2.internal - no node group config I0514 08:53:03.149516 1 utils.go:407] Skipping ip-10-0-0-101.ec2.internal - node group min size reached I0514 08:53:03.149523 1 utils.go:407] Skipping ip-10-0-0-54.ec2.internal - node group min size reached I0514 08:53:03.149624 1 static_autoscaler.go:309] Scale down status: unneededOnly=true lastScaleUpTime=2018-05-14 08:50:38.709254596 +0000 UTC lastScaleDownDeleteTime=2018-05-14 08:24:44.463804024 +0000 UTC lastScaleDownFailTime=2018-05-14 08:24:44.46380406 +0000 UTC schedulablePodsPresent=false isDeleteInProgress=false

one of worked node created as autoscalling: Kubelet Version: v1.9.3 Kube-Proxy Version: v1.9.3 ExternalID: i-024a88c5e2b6a0a50 Non-terminated Pods: (1 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits

kube-system kube-proxy-qtssq 0 (0%) 0 (0%) 0 (0%) 0 (0%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) CPU Requests CPU Limits Memory Requests Memory Limits

0 (0%) 0 (0%) 0 (0%) 0 (0%) Events:

Am I missing some config?

jorge07 commented 6 years ago

Kube-aws version?

nandakishorebhandari commented 6 years ago

I have used latest release,

kube-aws version a55c6657783c80f254b3be23ba0cd2ad45e811d7

jorge07 commented 6 years ago

If you see the full logs, I think you should have some kube-system pods running in your workers. Take a look at where are the rescheduler, the autoscaler and the dashboard for example.

nandakishorebhandari commented 6 years ago

ok, will check that, by the way, i have enabled autoscalling on running cluster using kube-aws update. Will check it with fresh cluster.