search

tiddlyhost / tiddlyhost-com

Rails application for creating and hosting TiddlyWiki sites, plus resources for deploying it to https://tiddlyhost.com/
Other
184 stars 17 forks source link

The S3 backup script should be smart enough to not download the same key again and again #163

Closed simonbaird closed 3 weeks ago

simonbaird commented 3 years ago

Currently it does a full download of the entire bucket, which is a: inefficient and b: depending on the frequency, it's likely to be a significant contribution to my S3 bandwidth costs

It should be possible to skip the blobs that are already backed up.

simonbaird commented 3 years ago

Maybe this is the way to do it: https://rclone.org/ .

simonbaird commented 2 years ago

Probably don't need rclone since aws s3 sync can do the same thing..

simonbaird commented 3 weeks ago

Need to rethink S3 backups now that they're larger. This is out of date I think.