Setup on the alternative bastion (`bastion-alt.gigadb.host`) a daily `rclone sync` job for keeping user dropboxes in sync between live and alternative

gigascience / gigadb-website

Source code for running GigaDB

http://gigadb.org

GNU General Public License v3.0

9 stars 15 forks source link

Setup on the alternative bastion (`bastion-alt.gigadb.host`) a daily `rclone sync` job for keeping user dropboxes in sync between live and alternative #2000

Open rija opened 3 months ago

rija commented 3 months ago

User Story

As a curator I want to ensure the user dropboxes area is backed up So that I have piece of mind when I work on files on the dropboxes

Acceptance criteria

Given there is a dataset being curated When there are new files added then the backup should have the new files

Given there is a dataset being curated When there are new files modified then the backup should have the updated files

Given there is a dataset being curated When files are deleted from the user dropbox by the curators then the backup should also delete the files

Additional infos

sync files from the live infrastructure into the alternative infrastructure.
the process runs on the alternative infrastructure
runs daily
rclone sync does the sync
bats test to test the correctness of the configuration

Tasks

[x] #2004
[x] #2005
[x] #2006
[x] #2007

rija commented 3 months ago

Hi @pli888, @kencho51,

the command I used to do the manual sync:

$ rclone --config rclone_upstream.conf -v  sync live-bastion:/share/dropbox/ /share/dropbox/

and the config used for it, rclone_upstream.conf looks like this:

[live-bastion]
type = sftp
host = bastion.gigadb.host
user = centos
key_file = /home/centos/.ssh/upstream_from_alt-live
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

The public part of that ssh key was manually added to the current production bastion's authorized_key file.

I'll update the various tasks with more details based on the above.