search

bustle / shep

A framework for building JavaScript Applications with AWS API Gateway and Lambda
MIT License
378 stars 27 forks source link

Config sync hits some amazon limit #254

Open chris-olszewski opened 7 years ago

chris-olszewski commented 7 years ago

I have a project with 12 functions and I get a TooManyRequestsException: Rate exceeded on a bunch of the Lambda API calls. Easy fix is to turn down the concurrency on the Promise.map call.

chris-olszewski commented 7 years ago

From amazon:

- Describe actions, such as ListAliases, ListFunctions etc -- these functions simply retrieve cached data, so they have the highest retry limits. - Modify actions, such as CreateFunction, CreateAlias, etc -- these functions create or modify resources so they have a lower request limit than describe actions.

They suggested to implement retry exponential retry behavior like this

reconbot commented 7 years ago

That's a horrible code example, the JS SDK support auto retries for some of their products. Maybe it needs to be enabled?

On Wed, Jun 14, 2017 at 11:19 AM, Chris Olszewski notifications@github.com wrote:

From amazon:

  • Describe actions, such as ListAliases, ListFunctions etc -- these functions simply retrieve cached data, so they have the highest retry limits.
  • Modify actions, such as CreateFunction, CreateAlias, etc -- these functions create or modify resources so they have a lower request limit than describe actions.

They suggested to implement retry exponential retry behavior like this http://docs.aws.amazon.com/general/latest/gr/api-retries.html

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bustle/shep/issues/254#issuecomment-308465057, or mute the thread https://github.com/notifications/unsubscribe-auth/AABlbkAmp-seRyUKGMzClpxvnNee9dTpks5sD_nugaJpZM4N40Gw .

-- Francis Gulotta Director of Engineering Bustle.com / @bustlelabs

chris-olszewski commented 7 years ago

I've been playing around with maxRetries and retryDelayOptions in AWS.config seems to help slightly docs. I can get the concurrency up to 3 without errors now (up from 2), but this requires fairly insane backoff options of allowing 10 retries with each retry being delayed one second per retry attempt. This actually slows down the sync.

reconbot commented 7 years ago

What about no concurrency? Since this is probably account wide and not per user we might want to be conservative.

ndastur commented 6 years ago

Hi guys, is there a fix for this. Bit of a blocker for shep if maximum number of functions is limited to 10 or so??