Lambda Parallelization Factor

awslabs / amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.

Apache License 2.0

338 stars 95 forks source link

Lambda Parallelization Factor #84

Closed lammedon closed 3 years ago

lammedon commented 4 years ago

We use Lambda functions fed by Kinesis in our architecture.

This app monitors throughput for Kinesis (number of records, amount of data) via CloudWatch. In addition to this it would be good to monitor lambda performance via the CloudWatch metric IteratorAge to scale the relatively new parameter in Kinesis/Lamba, "parallelization factor".

https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-parallelization-factor-for-kinesis-and-dynamodb-event-sources/

IanMeyers commented 4 years ago

That does sound useful. Will look into it and see how hard it will be to do. Would need to bound to a single application however, which could be unpredictable.

IanMeyers commented 3 years ago

Closing as duplicate of #63