search

wustl-oncology / analysis-wdls

Scalable genomic analysis pipelines, written in WDL
MIT License
5 stars 11 forks source link

changing localization_optional to false in mutect tool #139

Closed evelyn-schmidt closed 2 weeks ago

evelyn-schmidt commented 10 months ago

Change localization_optional:true to false.

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/mutect.wdl#L19-L22

malachig commented 10 months ago

Just to document the rationale here. The optional localization (streaming the input BAMs from a cloud bucket) works almost all the time. But we have hit a case where it fails consistently on a single shard causing the whole workflow to fail. This is similar to the behavior we saw with VarScan where we were not able to understand why.

For our current use cases the inconvenience of a workflow run failure is high. We have a small number of cases but all require clinically compatible turnaround times. The increased robustness of localizing the inputs outweighs the cost savings for our application.

evelyn-schmidt commented 6 months ago

Attempting to formalize documentation of pipeline releases, we have chosen some sample datasets to re-run through releases of the pipeline. Currently, I am testing v1.1.4 using the pancreas trail case 5120-17. This is the case that originally inspired this PR because of its strange failure on shard-3 of Mutect. My re-run also resulted in the same failure (since this PR was not merged). I am rerunning case 5120-17 with these fixes to confirm that turning off localization_optional resolves this issue. If it does, then we should consider turning off this setting in other parts of the pipeline, for example, VarScan.

evelyn-schmidt commented 6 months ago

Confirmed that this did fix the problem. Include in next formal release.

Files where localization_optional:true. @malachig maybe we should changed these all to false?

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/select_variants.wdl#L22-L23

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/variants_to_table.wdl#L15-L16

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/downsample.wdl#L15

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/samtools_flagstat.wdl#L10-L11

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/bam_to_cram.wdl#L12

https://github.com/wustl-oncology/analysis-wdls/blob/c89037d4164215c3d73f8fd766f63c92933dcfc2/definitions/tools/docm_gatk_haplotype_caller.wdl#L18-L21

malachig commented 6 months ago

@evelyn-schmidt it looks like these additional places do not involve scattered tasks? So we are probably less likely to encounter problems with those. If so, I think we can leave them for now as we haven't noticed any problems to date.

malachig commented 2 weeks ago

Just adding a comment here that I also encountered an error on shard-3 of a completely different case. Exit code 247.

Required file output '/cromwell_root/mutect.filtered.vcf.gz' does not exist.

malachig commented 2 weeks ago

And once again, applying this fix seems to have resolved the issue.