search

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
901 stars 240 forks source link

Add FTP support to `AbstractJobStore._get_size` #5134

Closed stxue1 closed 3 days ago

stxue1 commented 1 month ago

_get_size currently doesn't support FTP as it requires special commands and wizardry.

This will also need to support extracting username/password from the URL and/or supporting environmental variables to override the username/password

This will allow FTP files to work with #5133

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1660

mr-c commented 1 month ago

I implemented the equivalent of this in https://github.com/ohsu-comp-bio/cwl-tes/blob/03f0096f9fae8acd527687d3460a726e09190c3a/cwl_tes/ftp.py#L231 Feel free to steal!