sbstjn / serverless-dynamodb-autoscaling

Serverless Plugin for Amazon DynamoDB Auto Scaling configuration.
https://sbstjn.com/serverless-dynamodb-auto-scaling-with-cloudformation.html
MIT License
148 stars 27 forks source link

Rate Exceeded error when adding scaling to preexisting stack. #35

Open cdichiara opened 6 years ago

cdichiara commented 6 years ago

Hi,

Nice plugin, by the way ... really appreciate that you made it.

I think Issue #3 might still be a problem for preexisting stacks where scaling is added: https://forum.serverless.com/t/autoscaling-for-amazon-dynamodb/2071/15?u=cdichiara I suspect that you don't have prolific enough dependencies to protect against updates to preexisting stacks:

          "irhbackendAutoScalingTargetWriteDatabaseStageRegion": {
            "DependsOn": [
              "UsersDb",
              "irhbackendDynamoDBAutoscaleRoleDatabaseStageRegion"
            ],

These are logical dependencies ... to protect against rate exceeding I think you're going to want illogical dependencies that serialize the creation of scaling policies and scaling targets, or whatever's adding up to whatever limit it's triggering. So, in other words, something like this:

          "irhbackendAutoScalingTargetWriteDatabaseStageRegion": {
            "DependsOn": [
              "UsersDb",
              "irhbackendDynamoDBAutoscaleRoleDatabaseStageRegion",
              "irhbackendAutoScalingTargetWriteLastDatabaseStageRegion"
            ],

It works for you when you do the initial install because the creations must be serialized by the logical dependencies, somehow.

I'll try to get around to a possible fix in a pull request, but I can't do it now ... my workaround is basically adding autoscaling to one database at a time, up to now it hasn't triggered the rate issue, but it's ponderous.

Thanks!

bsdkurt commented 6 years ago

After manually adding auto-scaling to my existing tables and hitting the rate limit problem, I can confirm the current DependsOn implementation in the plugin will not limit the rate properly and will get Rate exceeded when adding or changing many AWS::ApplicationAutoScaling::ScalingPolicy resources when the tables already exist and may also get it if many ScalingPolicy are changed in the same update.

DependsOn is the proper solution, but what it depends on is wrong. In order to eliminate the Rate exceeded error in all cases, the DependsOn for AWS::ApplicationAutoScaling::ScalingPolicy should be the previous ScalingPolicy and nothing else. This will serialize additions and changes to the policies when creating or changing them regardless if the tables were previously created or not.

This solution is similar to how AWS explains that to avoid table related rate limits, place DependsOn on your table to depend on the previous table.