DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
900 stars 240 forks source link

Add FTP support to jobstores #5142

Closed stxue1 closed 6 days ago

stxue1 commented 1 month ago

Closes #5134

Also turns all url_exists requests and size requests to HEAD requests instead, as mentioned here: https://github.com/DataBiosphere/toil/issues/5114#issuecomment-2392527695

In my local testing, this significantly improves FTP performance. (Previously an existence check on a small text file on an insecure FTP server took around 5 seconds, though I'm unsure why it would take so long originally)

Changelog Entry

To be copied to the draft changelog by merger:

Reviewer Checklist

Merger Checklist