search

awslabs / amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Apache License 2.0
338 stars 95 forks source link

can't scale up from minimum #18

Closed szinck closed 9 years ago

szinck commented 9 years ago

If you create a stream having $minShards, auto-scaling will not scale up even though it thinks it needs to.

I think this is due to the short circuit logic here:

https://github.com/awslabs/amazon-kinesis-scaling-utils/blob/master/src/main/java/com/amazonaws/services/kinesis/scaling/StreamScaler.java#L342-L354

Basically if numberOfShards == minShards, it will not scale up. It also looks like it will not scale down once it reaches maxShards.

Here are some logs from where I tested this:

19-May-2015 22:35:43.203 INFO [pool-1-thread-1] com.amazonaws.services.kinesis.scaling.auto.StreamMonitor.processCloudwatchMetrics Stream Used Capacity 92.82% (973,236 Bytes of 1,048,576)
19-May-2015 22:35:43.203 INFO [pool-1-thread-1] com.amazonaws.services.kinesis.scaling.auto.StreamMonitor.processCloudwatchMetrics Scale Up Stream test-stream by 2 Shards as PUT has been above 85% for 1 Minute
s
19-May-2015 22:35:43.469 INFO [pool-1-thread-1] com.amazonaws.services.kinesis.scaling.StreamScaler.scaleStream Scaling Stream test-stream from 1 Shards to 3
19-May-2015 22:35:43.686 INFO [pool-1-thread-1] com.amazonaws.services.kinesis.scaling.StreamScaler.scaleStream Minimum Shard Count of 1 Reached
IanMeyers commented 9 years ago

Ah sorry about this - the integration test made an incorrect assumption. Fixed in place with a patch to .9.1.8

szinck commented 9 years ago

wow thanks!