Backup /usr/caltech_curation_files

[valearna]

Set up duplicity with Raymond to back up curator files and scripts (keep symlinks)

Also need a place for files that shouldn't get backed up.

We need to talk to Wen to figure out what to back up from her files

[valearna]

From 05/08 meeting with Raymond:

Restic seems a great option and it has plenty of config parameters. We can use it to backup data to s3

Backup policy:

• create daily snapshots up to 30 days
• create monthly snapshots up to 24 months

Avoid encryption if we can - so that we can access the files even if we lose the keys

Examples:

restic -r s3:s3.amazonaws.com/daily_restic_backup init restic -r s3:s3.amazonaws.com/monthly_restic_backup init

restic -r s3:s3.amazonaws.com/daily_restic_backup /usr/caltech_curation_files --exclude-file=excludes.txt restic -r s3:s3.amazonaws.com/monthly_restic_backup /usr/caltech_curation_files --exclude-file=excludes.txt

restic forget -r s3:s3.amazonaws.com/daily_restic_backup --keep-last 30 --prune restic forget -r s3:s3.amazonaws.com/monthly_restic_backup --keep-last 24 --prune

Exclusion list:

/usr/caltech_curation_files/kimberly/uniprot/uniprot_paper_data/files/

Check with Wen about simplemine files

Restic has options to migrate from one repo to another (for example in case we need to move from s3 to a physical server in the future)

TODO:

• check if we can recover files from partial snapshots (in case the backup process does not end successfully)
• check if restic can preserve symlinks - YES (from the docs)
• check if it preserves modification dates - YES (tested)

[valearna]

Add documentation in form of a README file on github