search

BD2KGenomics / toil-lib

A common library for functions and tools used in toil-based pipelines
Apache License 2.0
4 stars 7 forks source link

S3IOStore cannot transfer big files #81

Closed glennhickey closed 7 years ago

glennhickey commented 7 years ago

This is a dealbreaker for toil-vg. Am currently using this patch: https://github.com/glennhickey/toil-lib/commit/b38d55e9bd04fd95d04687bc41a952f7805ff907

Need to get this or something equivalent upstream (PR to follow).

Interesting note: I also tried using Toil's multipart uploading functionality. It died on a 30G file. Error here: https://github.com/BD2KGenomics/toil-vg/issues/110#issuecomment-276095314

glennhickey commented 7 years ago

Just noticed that toillib.py and IOStore are gone from repo.

glennhickey commented 7 years ago

Looks like I'm confused between this toil-lib and Charlie's toil-lib, which aren't related as I thought.