search

awslabs / amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Apache License 2.0
338 stars 95 forks source link

CloudwatchMetrics collection wrong #65

Closed kazysgurskas closed 5 years ago

kazysgurskas commented 6 years ago

Running autoscaling 9.5.5. It seems that StreamMonitor collects wrong data from CloudWatch. I'm seeing roughly 10x lower numbers in the logs:

INFO: Records: Stream raw Used PUT[Records] Capacity ~ 1.25% (25 Records of 2000)
INFO: Bytes: Stream raw Used PUT[Bytes] Capacity ~ 7.52% (158,772 Bytes of 2097152)

In CloudWatch dashboard under same timestamp I'm seeing 250 records and ~1.6 MB PUT capacity usage. This causes the application to scaledown aggressively.

IanMeyers commented 5 years ago

Is this still an issue? I've reviewed how this works and there's nothing obvious around an order of magnitude transcription error, so not sure why you'd see this. We take the sum of all the samples since we last checked, and then divide by the interval since we last checked, which gives us an average over the period. If this still an issue, can you drop detailed logs into the issue and this will show the interval being used?

kazysgurskas commented 5 years ago

Do not have a way to replicate this anymore, sorry :) Never got it to work, so quit trying. Thanks for the reply, though!