sebdah / dynamic-dynamodb

Dynamic DynamoDB provides auto scaling for AWS DynamoDB
http://dynamic-dynamodb.readthedocs.org/
Apache License 2.0
618 stars 124 forks source link

Package for Lambda #280

Open bradmurray opened 9 years ago

bradmurray commented 9 years ago

It would be great if we could run this in Lambda now that it supports python and scheduling.

DanDiac commented 9 years ago

+1

bradmurray commented 9 years ago

I have actually written a version of this in node that I will post once it's cleaned up. I am running it in Lambda on a schedule and it seems to be working fine. Basically you just have to setup the function and create a role with the proper permissions and let it run.

I also want to set it up to allow for multiple tables in the config so that one job can manage all of your tables and indexes.

tmitchel2 commented 8 years ago

A lambda version of this would be awesome

mbravorus commented 8 years ago

+1

I wonder if it would be possible to run it statelessly, without scheduling, reacting to threshold alerts on Dynamo metrics and/or circuit breaker availability via Cloudwatch?

bradmurray commented 8 years ago

You need a schedule to handle reductions since there is not a trigger that will indicate under utilization, but that can run once an hour or even less since you can only reduce 4x per day. I make my scaler only reduce a max of once per 6 hours, but if it is in the second 6 hour block of the day (UTC) and it hasn't reduced it will reduce more often so that it hopefully uses them all up by the end of the day.

I have mine one every 5 minutes and also on an SNS when my read throttles are over 2 for 2 consecutive minutes. I am still fine-tuning the logic, but my last 24 an auto pilot looks like this:

dynamo-history

My spikes are surges from search engines.

tmitchel2 commented 8 years ago

I'm setting it up to run every minute on lambda.... I'm using the runonce option, set the cloudwatch data window to -1 minute and for a minute duration, set the check-interval to zero to avoid the sleep at the end, log-file off. Even running every minute it uses barely any lambda resource however Im not using this in anger just yet, mainly because my usage is very bursty at the moment and it can't keep up.

alessandro-lac commented 8 years ago

+1 for the Lambda version :)

uded commented 8 years ago

A Lambda package would make this complete and easy to use. Looks not a like mission impossible - package for Lambda deployment, configuration via S3 bucket...

sebdah commented 8 years ago

A Lambda version of Dynamic DynamoDB would be really cool. PRs for this is more than welcome.

bradmurray commented 8 years ago

https://github.com/bradmurray/DynamoDBLambdaMonitor

mtngld commented 8 years ago

Another good lambda version: https://github.com/rockeee/dynamic-dynamodb-lambda

tmitchel2 commented 8 years ago

I've just released my version here: https://github.com/channl/dynamodb-lambda-autoscale

angulito commented 8 years ago

I've added new features to the dynamic-dynamodb-lambda developed by rockeee. New features:

You can download this lambda from here: https://github.com/touchvie/dynamic-dynamodb-lambda

I hope that this lambda can help you. It works fine.