Support signed URLs in GATK tools

broadinstitute / gatk

Official code repository for GATK versions 4 and up

https://software.broadinstitute.org/gatk

Other

1.68k stars 588 forks source link

Open droazen opened 4 years ago

droazen commented 4 years ago

Requested by @davidbenjamin and @abaumann

We'll need to figure out what to do about "companion" files such as indices and sequence dictionaries when given a signed URL to the main file.

lbergelson commented 3 years ago

Initial signed url support exists as of https://github.com/broadinstitute/gatk/pull/6526 but there are some caveats that may prevent production use.

Some methods in the Path api weren't implemented yet, the noticeable exception is the method used when finding sister paths such as the .bai. This means that indexes must be manually specified or you'll hit a UnimplementedFeatureException.
There isn't retry logic so http failures will cause job failure.