search

openaq / openaq-ingestor

1 stars 1 forks source link

add limit to concurrent executions to prevent overrun #12

Open russbiggs opened 2 weeks ago

russbiggs commented 2 weeks ago

Currently if the ingestor function take a long time to run (due to an error in the pipeline) there is no limit on the number of concurrent executions, mean the function can scale up and cost a lot.

Configuring a reserved concurrency limit in the CDK configuration should solve this. Currently the function hovers around 30-40 concurrent executions so limiting to 100 would provide a safe ceiling.

russbiggs commented 2 weeks ago

The reserved_concurrent_executions seems to be the parameter to set in the Function construct.

https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_lambda/Function.html