The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Apache License 2.0
338
stars
95
forks
source link
Switching to ListShards and DescribeStreamSummary #69
This change is a migration to use ListShards for shard discovery, and DescribeStreamSummary for stream status discovery. ListShards offers much higher TPS and throttles at a stream level, so this should help with customers hitting their DescribeStream limits when scaling up large streams.
Tested it with a personal stream, seems to work fine. Sample logs below for scaling a stream from 1 shard to 11.
This change is a migration to use ListShards for shard discovery, and DescribeStreamSummary for stream status discovery. ListShards offers much higher TPS and throttles at a stream level, so this should help with customers hitting their DescribeStream limits when scaling up large streams.
Tested it with a personal stream, seems to work fine. Sample logs below for scaling a stream from 1 shard to 11.
Scaling Operation Complete Scaling Direction: UP Shard shardId-000000000001 - Start: 0, End: 30934760629176224260563479694650258734, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000003 - Start: 30934760629176224260563479694650258735, End: 61869521258352448521126959389300517469, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000005 - Start: 61869521258352448521126959389300517470, End: 92804281887528672781690439083950776204, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000007 - Start: 92804281887528672781690439083950776205, End: 123739042516704897042253918778601034939, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000009 - Start: 123739042516704897042253918778601034940, End: 154673803145881121302817398473251293674, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000011 - Start: 154673803145881121302817398473251293675, End: 185608563775057345563380878167901552409, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000013 - Start: 185608563775057345563380878167901552410, End: 216543324404233569823944357862551811144, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000015 - Start: 216543324404233569823944357862551811145, End: 247478085033409794084507837557202069879, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000017 - Start: 247478085033409794084507837557202069880, End: 278412845662586018345071317251852328614, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000019 - Start: 278412845662586018345071317251852328615, End: 309347606291762242605634796946502587349, Keyspace Width: 30934760629176224260563479694650258734 (9.091%) Shard shardId-000000000020 - Start: 309347606291762242605634796946502587350, End: 340282366920938463463374607431768211455, Keyspace Width: 30934760629176220857739810485265624105 (9.091%)