search

sebdah / dynamic-dynamodb

Dynamic DynamoDB provides auto scaling for AWS DynamoDB
http://dynamic-dynamodb.readthedocs.org/
Apache License 2.0
618 stars 124 forks source link

Handling momentary bursts of read/write #332

Closed smartinsert closed 7 years ago

smartinsert commented 7 years ago

Thank you for making our lives a little easier by developing the dynamoDB autoscaling tool.

I have a data store running on another server that populates DynamoDB through a watcher process. Sometimes there are situations when we might need to offload loads of data from the data store through this process onto dynamoDB, although your tool works like charm under normal circumstances, I was wondering if there is any time remedy to sudden bursts of writes/reads on dynamoDB which might go unnoticed in the tool given the smallest polling interval.

For example in the below snapshot, the writes shot up from 0 to 34.15 within a second. dynamodb_writes

Given that one has around 20 tables and info log mode. The log files can become really big over an entire day.

My questions here are:

  1. Is there any way to manage burst of read/writes using this tool ?

  2. How to control the logging in info mode ?

  3. How to have separate configuration files to deal with different tables to fine tune the provision on a table level ?

sebdah commented 7 years ago

  1. We cannot handle the bursts in a good way, since we can only retrieve data from CloudWatch every 60 seconds.

  2. Check this out: http://dynamic-dynamodb.readthedocs.io/en/latest/configuration_options.html#logging-configuration. You could write your own Python logging configuration file and use log-config-file to point to that file.

  3. Why do you need separate configuration files? In any case, if you do, just start two dymamic-dynamodbs with the different configuration files as input parameters. Only one config file is supported by one dynamic-dynamodb instance.