aws-cloudformation / rain

A development workflow tool for working with AWS CloudFormation.
Apache License 2.0
812 stars 73 forks source link

Handle throttling in cc deploy #242

Open ericzbeard opened 10 months ago

ericzbeard commented 10 months ago

Services will throttle too many parallel requests to create resources. In cc deploy, all resources at the same level in the dependency graph get deployed at the same time, with no retries. We need to implement exponential backoff for RateExceeded or ThrottlingException errors, and batch requests if there are more than 5 going to same service at the same time.

PatMyron commented 3 months ago

rain forecast --include-iam permission checks get throttled too:

$ rain forecast --experimental cfn.yaml --include-iam
FG002 FAIL on line 9: AWS::S3::Bucket Bucket - Insufficient permissions to create arn:aws:s3:::rain-XXX
    operation error IAM: SimulatePrincipalPolicy, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: XXX, api error Throttling: Rate exceeded
ericzbeard commented 3 months ago

rain forecast --include-iam permission checks can get throttled too:

Yeah that feature needs to be redesigned to take advantage of batching calls to the policy simulator, but the way the actions get batched are not consistent per resource, and not documented, so it's a trial and error approach.