Backup S3 production bucket periodically

EBWiki / EBWiki

repository of police abuse cases against people of color

https://ebwiki.org

Apache License 2.0

83 stars 106 forks source link

Backup S3 production bucket periodically #713

Open campgurus opened 8 years ago

campgurus commented 8 years ago

We need to figure out a good way to backup our uploaded images bucket.

Here is a good discussion of the alternative approaches: http://stackoverflow.com/questions/17832860/backup-strategies-for-aws-s3-bucket

copying to another s3 bucket would seem to meet our needs for protection. It does make sense to think of what exactly we are protecting ourselves from. eg. backing up to another s3 bucket will not protect us against a general failure of AWS or S3.

campgurus commented 8 years ago

I set up a backup bucket called blackops-backup, but have not automated the process yet.

trystant commented 8 years ago

What is the manual version of the backup process and can we place that on the wiki (for the wiki; ha!)

campgurus commented 8 years ago

Added to the wiki

rlgreen91 commented 6 years ago

@campgurus @trystant So are we going into the production console and running this command? Writing a rake task or background job that does this would be good for Summer of Code people.

trystant commented 6 years ago

we're not...this should be a good SoC project.

gt50 commented 5 years ago

Regarding the backup, s3 is very durable. If you have versioning turned on, it is even better. Do you have another cloud that you would like to hold these in? Backing up to a second s3 bucket with aws s3 sync would help against accidental or intentional deletions. If the fear is complete aws meltdown, then you would need to run the sync to a local vm on another cloud provider to keep a copy of the data.

trystant commented 5 years ago

I agree that S3 is very durable and it would still be helpful to:

create or use a program that would sync to a backup s3 bucket. Could be a rake task
create or use a program that would sync to another cloud backup solution (digital ocean, cloudinary, etc)
create or use a program to schedule the runs of these other tasks.

Care to evaluate our S3 bucket @gt50?

gt50 commented 5 years ago

No problem @trystant. Set up a u/p with access to the bucket in IAM, and I can take a look.

Depending upon the size of the bucket there are many ways to deal with this. A very small instance on another cloud could run an "aws s3 sync" command to keep a local copy of all of the data. This is probably the simplest option. Another option is using lambda to places copies of any file that is place in the bucket in other locations.

I just checked out DO's offerings. I think Spaces would be perfect for this.