gulp deploy is stopped with TooManyRequestsException

[ophilippsen]

When running "gulp deploy" to update a created stack, deployment stops due to a TooManyRequestsException.

e.g.

[2017-01-18T22:30:10.915Z] INFO: spacefinder/7637 on *.local: Created/updated Lambda function spacefinder-api-development-bookings-Delete

    at Request.<anonymous> (/Users/ophilippsen/Sources/awslabs/aws-serverless-auth-reference-app/api/node_modules/aws-sdk/lib/request.js:670:12)
  message: 'Rate exceeded',
  code: 'TooManyRequestsException',
  time: 2017-01-18T22:30:12.274Z,
  requestId: 'ad0a5be0-ddcd-11e6-8677-a198e9e95084',
  statusCode: 429,
  retryable: false,
  retryDelay: 87.86947171700652 }

[23:30:12] 'create_lambda_functions' errored after 2.28 s

As far i understand i ran into throttling - how can i increase allowed requests or overwrite sdk defaults to not run into this problem ?

[justonian]

Hi there-

Just wanted to check if you encountered this problem again or were able to move past it. I was not able to reproduce when I tried to undeploy and re-deploy. I suspect if you undeployed and re-deployed the project multiple times in a very short window of time you might encounter throttling exceptions from control plane calls to continually change service configuration.

Please let us know if this is still causing an issue for you.

Thanks,

Justin

[ophilippsen]

Hi Justin,

thanks for asking - got still the same problem, i´m deploying to eu-central and as far i figured out, it´s a throttling problem with the promise.all() implementation for the lambda updates from swagger (to many parallel requests).

Will adjust your implementation by using the bluebird lib and limit concurrency to solve the problem and tell you the results.

Greetings, Oliver

[ophilippsen]

Hi Justin,

finally the implementation with bluebird and limit the concurrent requests did fix the problem. i can provide a merge request, when you are interested.

Greetings, Oliver

[justonian]

Hi Oliver-

Thanks for the update. It'd be great if you can submit a pull request to show your changes. I can't guarantee we'd merge it, as is, but we will certainly dig through it to fully look at what you had to change to make this work properly for you and try to remedy the root cause.

Thanks again for your interest in the project.

Justin

[dscowan]

I'm seeing the same problem. It seems to happen once you add more api methods.

[justonian]

Hi dscowan- If you run gulp deploy again, the process is fully idempotent and it will almost certainly succeed on the 2nd attempt. I've only seen this issue once or twice and it was always transient in nature, depending on other activities which had been recently done in the account.

[dscowan]

Hi justonian, I did try that three times and failed at the same spot every time. It seems once I added 2 more api methods this happened. I then removed them and was able to deploy. Then to test I added back and they failed. Then removed some other existing methods and was able to deploy so I am pretty confident this is the issue.