FredHutch / lf-backup

lf-backup (aka large file backup) takes a list of filenames from a csv file or sql and copies the files to a swift objectstore
Other
0 stars 0 forks source link

Restore expects directory structure to already exist, if not it fails. #10

Closed robert-mcdermott closed 7 years ago

robert-mcdermott commented 7 years ago

I tested the restore feature and it works, but it assumes that the directory structure above the files being restored already exist:

$ lf-backup -C lf-backup4 -r 7 --prefix /data                                                                            
restoring container lf-backup4
restoring file fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/PCx_141.bam
Error downloading object 'lf-backup4/fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/PCx_141.bam': [Errno 2] No such file or directory: '/data/fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/PCx_141.bam'

If I make the expected directory structure ahead of time:

$ mkdir -p /data/fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/

Then the restore will work:

$ lf-backup -C lf-backup4 -r 7 --prefix /data                                                                 
restoring container lf-backup4
restoring file fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/PCx_141.bam
fh/fast/nelson_p/Ilsa/RNAseq/Xenografts/BAM/PCx_141.bam [auth 0.000s, headers 0.174s, total 179.696s, 21.910 MB/s]

If we are doing restore from a complete system failure, it's possible that the tape restore will have not finished yet (and we don't want to wait for it) before the lf-backup restore starts restoring files.

jmkatcher commented 7 years ago

Thanks for noticing that! I'll add a check/pre-create of the necessary directory structure and close this when it's checked in.

jmkatcher commented 7 years ago

Added and checked in creation of restore destination path for non-null destination paths.