Gsutil parallel composite upload temp files being written to GCS

va-big-data-genomics / trellis-mvp-functions

Trellis serverless data management framework for variant calling of VA MVP whole-genome sequencing data.

6 stars 1 forks source link

Gsutil parallel composite upload temp files being written to GCS #8

Open pbilling opened 4 years ago

pbilling commented 4 years ago

After switching from Cromwell:46 to 47, observed that temp files are being written to cloud storage. This has had the side effect of generating errors when Trellis tries to add these files to the database, because it looks for a "Blob" label that does not exist, because these filenames don't match the pattern for adding the "Blob" label.

pbilling commented 4 years ago

Checking the stackdriver logs, looks like additional temp files were created in GCS that no longer exist. Wondering if these are deleted after a cromwell workflow is completed?