Too Many Requests (Service: ApiGateway, Status Code: 429, ...) from AWS::ApiGateway::Model

[MaximilianSchon]

Name of the resource

AWS::ApiGateway::Model

Resource Name

AWS::ApiGateway::Model

Issue Description

When creating/updating many models (I currently have 114 models of varying sizes, but this error started happening before, at varying intervals) deployment fails with the error:

Too Many Requests (Service: ApiGateway, Status Code: 429, ...)

Expected Behavior

It should create the model successfully. For example using exponential backoff to retry the create requests.

Observed Behavior

The deployment fails with the error:

Too Many Requests (Service: ApiGateway, Status Code: 429, ...)

Test Cases

Create tons of API Gateway models, the deployment should not fail. The same should go with updating and removing them.

Other Details

588 should have solved this.

[heshamnaim]

I am seeing this issue as well when using CDK to deploy many models

[akum32]

+1

[jweyrich]

We're facing the same problem when deploying some bigger projects. When there are many resources to be deployed to API Gateway, CloudFormation fails after hitting the hard limit of 5 reqs/second per account invoking CreateResource (ref).

[oanhhuynhpositive]

Any workaround for this yet friends?

[ronny]

One workaround is to force the resources to be created in serial instead of in parallel, by making them depend on each other.

With CDK, you could do something like this for each apigateway.Model and apigateway.Method:

const model = new apigateway.Model(...)
model.node.addDependency(otherModel)

const method = new apigateway.Method(...) // or `resource.addMethod` or `resource.addCorsPreflight`
method.node.addDependency(otherMethod)

[hervenivon]

One workaround is to force the resources to be created in serial instead of in parallel, by making them depend on each other.

With CDK, you could do something like this for each apigateway.Model and apigateway.Method:

const model = new apigateway.Model(...)
model.node.addDependency(otherModel)

const method = new apigateway.Method(...) // or `resource.addMethod` or `resource.addCorsPreflight`
method.node.addDependency(otherMethod)

This is the only workaround we found. We also got the surprise of being unable to deploy an API stack from scratch while it was ok iteratively: same error "Too Many Requests (Service: ApiGateway, Status Code: 429, Request ID: xxxxx)"

I ended up storing all methods created from new apigateway.Method(...) in arrays and build a dependency graph automatically.

The CloudFormation change set takes longer to compute, but it works like a charm.

I applied the exact same strategy to CfnDocumentationPart

[nickvasic]

Hi, has there been any progress on this issue? We are facing the same problem and had to remove the models from the API Gateway which is not the best practice.

[Zhorky]

I'll join into this conversation: We basically have the same issue. We just added CORS and everything started breaking on our 20-30 endpoint API. With response models, request models, authorizer etc. we are at about 150 calls to ApiGateway which is too much.

on our side it looks a bit like this;

last_method = None

for endpoint in endpoints:
    method = function_that_transforms_endpoint_data_to_method(endpoint) # type is apigateway.Method
    if last_method:
        last_method.node.add_dependency(method)
    last_method = method

Sadly this doesn't work at all and we still get "Too Many Requests (Service: ApiGateway, Status Code: 429, Request ID: xxxxx)" Something like this https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/589 would be appreciated.

[PeterBaker0]

Also adding my voice here - we use a DevOps approach utilising CDK in which Stacks are constantly being turned over (created, updated, destroyed) etc. During the destroy operation, we consistently see API Gateway rate limits which result in a lengthy cloud formation failed destroy. In our workflow, we've had to instrument tools to automatically retry the destroy operation many times to finally get success (sometimes it can take 3-4 attempts at destroying the stack).

Movement on this would be fantastic - very frustrating user experience.

[hannes-intenics]

I'll join into this conversation: We basically have the same issue. We just added CORS and everything started breaking on our 20-30 endpoint API. With response models, request models, authorizer etc. we are at about 150 calls to ApiGateway which is too much.

on our side it looks a bit like this;

last_method = None

for endpoint in endpoints:
    method = function_that_transforms_endpoint_data_to_method(endpoint) # type is apigateway.Method
    if last_method:
        last_method.node.add_dependency(method)
    last_method = method

Sadly this doesn't work at all and we still get "Too Many Requests (Service: ApiGateway, Status Code: 429, Request ID: xxxxx)" Something like this #589 would be appreciated.

We basically have exactly the same issue. Cloudformation should know about those limits and wait for deployment until finished.

[lamontadams]

Seeing this as well with a ~40 endpoint Rest API. I don't want to be "that guy" but this is a great opportunity for CloudFormation to build some user goodwill - I wouldn't be using it at all if cdk weren't essentially married to it.

[DustyLamp]

For us, the add dependency workaround doesn't have a significant enough effect and we've spent hours every week for the last few months decoupling our infrastructure more and more, including creating dozens of stacks. Even though we have a very modular architecture, we still find that we have to arbitrarily separate it further.

We've tried to remove duplicate models from being created across multiple methods, and we're still running into these 429 Too Many requests issues unless we deploy our api infrastructure piece by piece slowly which takes a long time.

Would be fantastic to see movement on this as it's crippling us to the point where we're looking for alternatives to the AWS CDK