Set up DynamoDB and S3 backups

bnolan / echt-server

Facial recognition social network

1 stars 0 forks source link

Set up DynamoDB and S3 backups #19

Open chillu opened 7 years ago

chillu commented 7 years ago

While DynamoDB replicates into at least three nodes, it doesn't protect us from logic errors. Particularly with "delete photo" and "unfriend" actions, a logic error could overwrite or wipe out data accidentally.

DynamoDB: https://aws.amazon.com/blogs/aws/cross-region-import-and-export-of-dynamodb-tables/

S3: Probably just a matter of implementing bucket versioning

chillu commented 7 years ago

OK Data Pipelines is crazy expensive in its default config: 1 hour of EC2 c4.large plus EMR = ~0.12USD per run. And the pipeline is configured to run for one table only. https://aws.amazon.com/emr/pricing/

Looking into alternatives: http://stackoverflow.com/questions/18896329/export-data-from-dynamodb

chillu commented 7 years ago

https://github.com/Purple-Unicorns/DynamoDbBackUp looks like a winner. We'll need to enable streams on the tables and versions on the s3 bucket. DynamoDB itself doesn't have version tracking

chillu commented 7 years ago

More context: https://blogs.cornell.edu/cloudification/2016/01/26/backing-up-dynamodb/ and https://www.reddit.com/r/aws/comments/5l8qso/dynamodb_in_production_with_backups/