broadinstitute / viral-pipelines

viral-ngs: complete pipelines
Other
51 stars 28 forks source link

speed up data localization for certain tasks #519

Closed dpark01 closed 6 months ago

dpark01 commented 6 months ago

Per recent updates on Terra's side: https://support.terra.bio/hc/en-us/articles/22607611273115-February-1-2024

Request cpuPlatform: "Intel Ice Lake" in the runtime block for tasks that tend to localize very large input Files (kraken, demux, etc). Switches from gsutil to gcloud storage and supposedly sees a 10x speedup in data localization (from 1Gbps to 10Gbps).