Stackup should gracefully handle Rate Limiting

realestate-com-au / stackup

a simple CLI and Ruby API for AWS CloudFormation

MIT License

97 stars 34 forks source link

Stackup should gracefully handle Rate Limiting #55

Closed stevemac007 closed 6 years ago

stevemac007 commented 6 years ago

We use stackup for deploying some large stacks that can take an hour+ to deploy. With increased development effort on our AWS account we often run into AWS Rate Limit issues.

We are seeing ERROR: Rate exceeded failures on some of our longer running deployments.

INFO: [10:21:56] BastionStack - UPDATE_COMPLETE
INFO: [10:21:57] StagingStack - UPDATE_IN_PROGRESS
ERROR: Rate exceeded
make: *** [run_stackup] Error 1

mdub commented 6 years ago

@stevemac007: Stackup takes advantage of the incremental backoff built into aws-sdk-core. It retries 3 times by default, but you can increase the that limit via the --retry-limit option, or setting $AWS_API_RETRY_LIMIT.

stevemac007 commented 6 years ago

Awesome, will look into trying that param.

Would be great to call this out in the README

katherinelim commented 6 years ago

Updated README with retry limit tips in https://github.com/realestate-com-au/stackup/commit/fd18e6102d37eaa031b1a45e1a4ae85c9307147f