search

hjacobs / kube-aws-autoscaler

Simple, elastic Kubernetes cluster autoscaler for AWS Auto Scaling Groups
GNU General Public License v3.0
94 stars 21 forks source link

Reduce small fluctuations of node count #34

Open linki opened 7 years ago

linki commented 7 years ago

The autoscaler is great. However it can get into a state where it switches the desired node count back and forth due to small increase and decrease of workload in the cluster.

Let's assume cluster autoscaler inspects all pods and calculates a desired capacity of 12 nodes. Now let's assume a cron job spins up a pod every 5 minutes which would just be enough to have autoscaler calculate a desired node count of 13 nodes.

In that case autoscaler would constantly switch between 12 and 13 desired nodes because it sometimes inspects the cluster while the cron job is running and sometimes while it's not.

This could be the reason for the following behaviour:

2017-08-25 09:11:28,679 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            7.7    24486Mi         68
2017-08-25 09:11:28,679 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:          8.6    27135Mi         85
2017-08-25 09:11:28,679 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 09:11:28,679 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        3.7    22026Mi        262
2017-08-25 09:11:28,679 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:           10.3    33373Mi         81
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         11.6    36910Mi         99
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        4.9    28643Mi        359
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 5 nodes required (current: 5)
2017-08-25 09:21:54,386 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 09:21:54,387 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            8.8    26000Mi         65
2017-08-25 09:21:54,387 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:          9.9    28800Mi         82
2017-08-25 09:21:54,467 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 09:21:54,467 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.6    20512Mi        265
2017-08-25 09:21:54,467 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 09:21:54,467 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 09:21:54,467 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            7.7    24486Mi         68
2017-08-25 09:21:54,468 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:          8.6    27135Mi         85
2017-08-25 09:21:54,468 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 09:21:54,468 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        3.7    22026Mi        262
2017-08-25 09:21:54,468 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 09:29:07,973 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 13 to 12..
2017-08-25 09:30:10,635 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 12 to 13..
2017-08-25 09:32:13,877 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 09:32:13,877 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:           10.5    33573Mi         83
2017-08-25 09:32:13,877 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         11.7    37130Mi        101
2017-08-25 09:32:13,877 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        4.7    28443Mi        357
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 5 nodes required (current: 5)
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            8.7    25789Mi         64
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:          9.8    28568Mi         80
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.7    20723Mi        266
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            7.8    24697Mi         69
2017-08-25 09:32:13,878 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:          8.7    27367Mi         86
2017-08-25 09:32:13,879 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 09:32:13,879 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        3.6    21815Mi        261
2017-08-25 09:32:13,879 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 09:35:19,408 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:36:20,768 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:37:24,468 INFO: Scaling activity in progress for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:38:26,175 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 13 to 12..
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.3    32195Mi         68
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.4    35614Mi         85
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.1    14317Mi        262
2017-08-25 09:42:34,171 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            8.8    25939Mi         66
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:          9.9    28733Mi         83
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.6    20573Mi        264
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            9.2    26964Mi         94
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:         10.4    29860Mi        113
2017-08-25 09:42:34,172 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 09:42:34,173 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        2.2    19548Mi        236
2017-08-25 09:42:34,173 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 5)
2017-08-25 09:44:39,548 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 12 to 13..
2017-08-25 09:45:40,820 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:46:42,212 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:47:44,407 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:48:45,912 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:49:47,210 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:50:49,303 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:51:50,916 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:52:53,485 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 09:52:53,485 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.3    32195Mi         68
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.4    35614Mi         85
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.1    14317Mi        262
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            9.1    26259Mi         72
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:         10.3    29085Mi         89
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.3    20253Mi        258
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 5)
2017-08-25 09:52:53,486 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 09:52:53,487 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:           10.1    28983Mi        112
2017-08-25 09:52:53,567 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:         11.3    32081Mi        133
2017-08-25 09:52:53,567 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 09:52:53,567 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        1.3    17529Mi        218
2017-08-25 09:52:53,567 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 09:52:53,654 INFO: Scaling activity in progress for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:54:58,367 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 13 to 12..
2017-08-25 09:56:00,124 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 12 to 13..
2017-08-25 09:57:03,123 INFO: Scaling activity in progress for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 09:58:05,208 INFO: Scaling activity in progress for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 10:02:13,104 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.3    32195Mi         68
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.4    35614Mi         85
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.1    14317Mi        262
2017-08-25 10:03:15,079 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:           10.1    30087Mi         88
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:         11.3    33296Mi        107
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        1.3    16425Mi        242
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            9.9    26885Mi        111
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:         11.1    29774Mi        132
2017-08-25 10:03:15,080 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 10:03:15,081 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        1.5    19627Mi        219
2017-08-25 10:03:15,081 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 10:03:15,114 INFO: Some nodes are not ready in ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 10:04:17,665 INFO: Scaling activity in progress for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I, not scaling down from 13 to 12
2017-08-25 10:05:19,721 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 13 to 12..
2017-08-25 10:13:39,371 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 10:13:39,371 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.2    30147Mi         67
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.3    33362Mi         84
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.2    16365Mi        263
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            9.0    27987Mi         67
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:         10.1    30986Mi         84
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.4    18525Mi        263
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 10:13:39,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 10:13:39,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            7.9    24783Mi         70
2017-08-25 10:13:39,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:          8.9    27461Mi         87
2017-08-25 10:13:39,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 10:13:39,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        3.5    21729Mi        260
2017-08-25 10:13:39,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 10:23:59,480 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 10:23:59,480 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.2    30147Mi         67
2017-08-25 10:23:59,480 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.3    33362Mi         84
2017-08-25 10:23:59,480 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 10:23:59,480 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.2    16365Mi        263
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            9.2    28187Mi         69
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:         10.3    31206Mi         86
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.2    18325Mi        261
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            8.5    25983Mi         82
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:          9.5    28781Mi        100
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        2.9    20529Mi        248
2017-08-25 10:23:59,481 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 10:34:19,371 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a:                       CPU     MEMORY       PODS
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: requested:            9.2    30147Mi         67
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: with buffer:         10.3    33362Mi         84
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: weakest node:         3.8    15504Mi        110
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: overprovision:        2.2    16365Mi        263
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1a: => 4 nodes required (current: 4)
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b:                       CPU     MEMORY       PODS
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: requested:            9.4    28687Mi         74
2017-08-25 10:34:19,372 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: with buffer:         10.5    31756Mi         91
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: weakest node:         3.8    15504Mi        110
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: overprovision:        2.0    17825Mi        256
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1b: => 4 nodes required (current: 4)
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c:                       CPU     MEMORY       PODS
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: requested:            9.9    28894Mi        110
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: with buffer:         11.1    31983Mi        131
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: weakest node:         3.8    15504Mi        110
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: overprovision:        1.5    17618Mi        220
2017-08-25 10:34:19,373 INFO: kube-2-WorkerAutoScaling-XYPSFHVJA2I/eu-central-1c: => 4 nodes required (current: 4)
2017-08-25 10:36:23,113 INFO: Changing desired capacity for ASG kube-2-WorkerAutoScaling-XYPSFHVJA2I from 12 to 13..

However, since not strictly wrong, a constantly flipping node count seems unnecessary and could trigger superflous pod restarts of client applications.

linki commented 7 years ago

A fix would reduce the number of creating and terminating skippers for ingress in our setup which would be a way to reduce the occurrences of https://github.bus.zalan.do/teapot/issues/issues/568.

hjacobs commented 7 years ago

This is a duplicate of #1.

hjacobs commented 7 years ago

A cool off period as described in #28 could mitigate the observed behavior.