search

norment / tsd_issues

Repo to track issues with TSD as tickets
2 stars 0 forks source link

Reduce p33 impact on TSD backup system #33

Closed ofrei closed 3 years ago

ofrei commented 4 years ago

Currently we use a lot of backup storage on TSD.

We need it, for sure, on acquisition data such as EEG or MRI scans. However, I'm not sure if backup is important for derived data, e.g. UKB. It's probably OK to not have the backup. TSD HPC disks do have some redundancy (i.e. a failure of 1 disk won't cause a complete data loss).​

My suggestion is to move as much as possible to "nobackup". To avoid changing your scripts it's OK to create a symbolic link to the old location.

TSD confirmed that

We do not follow symbolic links in the backup process.

ofrei commented 4 years ago

To clarify, here is more info on backup system from TSD: https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/directories-files/backup/#toc1

Basically, TSD backup system skips all folders that end with "nobackup". TSD backup is by default enabled only on /durable. But, for NORMENT projects we have backup enabled on /cluster as well. However, we need to use it well.

Btw, having a symbolic link from /cluster to /durable won't work - this is not worth trying.

My suggestion is to consider moving /cluster/projects/p33/users/YOURUSER/SOMEDATA to /cluster/projects/p33/users/YOURUSER/nobackup/SOMEDATA and create a symbolic link in order to keep your scripts as today: /cluster/projects/p33/users/YOURUSER/SOMEDATA -> /cluster/projects/p33/users/YOURUSER/nobackup/SOMEDATA