18F / identity-analytics-etl

ETL and SQL scripts for Login.gov data warehouse and business intelligence

10 stars 6 forks source link

Need a Redshift Disaster Recovery Solution #185

Closed MacHu-GWU closed 5 years ago

MacHu-GWU commented 5 years ago

User story

Currently, we don't have any Redshift disaster recovery solution.

Notes

What is the value to the user in this story?

Allow us to recovery Redshift data quickly.

What are things we should consider when making this story

The solution can handler data volume 10 times larger than the current one.
Ideally, it can be recovered in several hours.

Acceptance Criteria

[ ] Solution Description is documented.
[ ] Solution can adapt magnitude-of-ten
[ ] Solution has been implemented and tested.

Tasks to complete the story

Similar to Acceptance Criteria

Definition of Done

[ ] Pull requests meet the technical definition of done
[ ] Usability tested

MacHu-GWU commented 5 years ago

Reference

Amazon Redshift FAQ, backup and restore: https://aws.amazon.com/redshift/faqs/#Backup_and_Restore
Working with Redshift snapshot: https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-snapshots.html

lauraGgit commented 5 years ago

@MacHu-GWU It looks like we do get a backup from Redshift- do you think we need more than this?

Amazon Redshift replicates all your data within your data warehouse cluster when it is loaded and also continuously backs up your data to S3. Amazon Redshift always attempts to maintain at least three copies of your data (the original and replica on the compute nodes and a backup in Amazon S3). Redshift can also asynchronously replicate your snapshots to S3 in another region for disaster recovery.

MacHu-GWU commented 5 years ago

@MacHu-GWU It looks like we do get a backup from Redshift- do you think we need more than this?

Amazon Redshift replicates all your data within your data warehouse cluster when it is loaded and also continuously backs up your data to S3. Amazon Redshift always attempts to maintain at least three copies of your data (the original and replica on the compute nodes and a backup in Amazon S3). Redshift can also asynchronously replicate your snapshots to S3 in another region for disaster recovery.

Since we have a 24-Hour snapshot backup with our Redshift, but haven't tried with a trail disaster scenario yet.

I assume that we should do something like, assuming the redshift is destroyed, how do we recover it in 1 day with scripts / some recovery operation in AWS console.

lauraGgit commented 5 years ago

I think we can hold off on that until the time comes, going to close it for now.