search

henriqueribeiro / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
10 stars 6 forks source link

revised localization functions to improve stability #16

Closed geertvandeweyer closed 2 years ago

geertvandeweyer commented 2 years ago

These commits fix the erros during data (de)localization seen under high load: 

download: s3://uza-cloud-wes-temp-001/scripts/43a217dffe4acc6da7d8e63243564f8b to tmp/tmp.EqG0A6RsE/batch-file-temp                                                                                                                               
*** LOCALIZING INPUTS ***                                                                                                                                                                                                                         download: s3://uza-cloud-wes-temp-001/wgs-tmp/WGS/054a14f1-27e4-4813-8758-dfebddedd6bb/call-cutadapt/shard-61/attempt-2/script to uza-cloud-wes-temp-001/wgs-tmp/WGS/054a14f1-27e4-4813-8758-dfebddedd6bb/call-cutadapt/shard-61/attempt-2/script
An error occurred (404) when calling the HeadObject operation: Not Found