search

sneumann / SemanticMetabolomics

A collection of converters and examples to show how Metabolomics can be a good citizen in Linked-Open-Data
GNU General Public License v3.0
5 stars 3 forks source link

Create partial FTP mirror of Metabolights #1

Closed sneumann closed 9 years ago

sneumann commented 10 years ago

Create a mirror of the publicly available ISAtab files from
ftp://ftp.ebi.ac.uk/pub/databases/metabolights/studies/public/

To avoid downloading Terabytes of data, just download

[isa]_*.txt and [am]_*.tsv and [am]_*.csv

Check https://ohse.de/uwe/ftpcopy.html to create the mirror http://manpages.ubuntu.com/manpages/trusty/man1/ftpcopy.1.html because that can then be used as cron job.

sneumann commented 10 years ago

Better:

  lftp -c mirror --no-empty-dirs --include="/[iasm]_.*\.[tc][xs][tv]$" ftp://ftp.ebi.ac.uk/pub/databases/metabolights/studies/public/
culpinnis commented 9 years ago

I think @chweber has done this? At least we have some metabolites files lying around. @chweber please give a comment and close this issue afterwards (if all necessary steps has been made). Also we need to know how the download was made. Please mail some kind of documentation to me. Cheers

sneumann commented 9 years ago

Metadata mirror can be created with lftp as shown above, and a copy is living in sneumann@balzer:/vol/metabolights/studies/public

sneumann commented 9 years ago

Just for reference: You can also use rsync -rlpt -v -z --delete -f'+ [iasm]_*.[tc][xs][tv]' -f'+ */' -f'- *' rsync.ebi.ac.uk::pub/databases/metabolights/studies/public/ ./metabolights

culpinnis commented 9 years ago

Did not mind that rsync can also handle ftp connections. Great! I will integrate this into the update script.

sneumann commented 9 years ago

It can't, but rsync.ebi.ac.uk is an rsync server of the FTP server content...

culpinnis commented 9 years ago

Ah, okay. Well, it solves the problem!