Closed mwildehahn closed 7 years ago
This works well for single tables, but if you're trying to snapshot multiple tables, it will bring up instances for each table, which sucks.
Should we pull this in, or wait and update for blueprint variables?
Mine as well just wait. This only works well for 1 or 2 tables, each time it runs for a table it brings up by default 3 instances to run an EMR job. I'm not sure if you can get an AWS data pipeline job to run on more than 1 table at a time. On Fri, Sep 2, 2016 at 10:52 AM Michael Barrett notifications@github.com wrote:
Should we pull this in, or wait and update for blueprint variables?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/remind101/stacker_blueprints/pull/21#issuecomment-244444004, or mute the thread https://github.com/notifications/unsubscribe-auth/AArAUQOCxpSGbUDCPekdOZ4nBZsotifjks5qmGJcgaJpZM4I05H4 .
Ok cool, I'll hold off.
Ok, I updated this to take an array of dynamodb tables so that it will bring up 1 EMR cluster and then run several EMR activities (1 per table you're backing up), which is significantly better than the first implementation which brought up 1 cluster per table.
Sample config would be:
- name: schema
class_path: stacker_blueprints.dynamodb.DynamoDB
variables:
Tables:
TestTable1:
AttributeDefinitions:
- AttributeName: created
AttributeType: S
KeySchema:
- AttributeName: created
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
TestTable2:
AttributeDefinitions:
- AttributeName: created
AttributeType: S
- AttributeName: test
AttributeType: S
KeySchema:
- AttributeName: created
KeyType: HASH
- AttributeName: test
KeyType: RANGE
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
- name: schemaSnapshots
class_path: stacker_blueprints.dynamodb.snapshot.Snapshot
variables:
Activate: true
PipelineLogUri: s3://stacker-mh-empire-dev/logs
ResourceRole: 'DataPipelineDefaultResourceRole'
Role: 'DataPipelineDefaultRole'
SchedulePeriod: '1 day'
ScheduleType: 'cron'
StartDateTime: '2016-10-17T02:00:00'
SnapshotConfigs:
- TableName: ${schema::TestTable1Name}
S3Output: s3://stacker-mh-empire-dev/backups/${schema::TestTable1Name}
- TableName: ${schema::TestTable2Name}
S3Output: s3://stacker-mh-empire-dev/backups/${schema::TestTable2Name}
Going to review some more, but wanted to make sure I submitted that.
The DynamoDB Snapshot blueprint configures an AWS Data Pipeline that uses an EMR cluster to export a DynamoDB table to an S3 Bucket.
Example config: