Out of disk error from FDA reports workflow

[malachig]

I encountered a failure at this step in a recent test: immuno -> call-generateFdaMetrics -> call-aligned_tumor_dna_cram_to_bam

Encountered this error:

2023/03/12 15:46:32 Running user action: docker run -v /mnt/local-disk:/cromwell_root --entrypoint=/bin/bash quay.io/biocontainers/samtools@sha256:141120f19f849b79e05ae2fac981383988445c373b8b5db7f3dd221179af382b /cromwell_root/script
[E::bgzf_flush] File write failed (wrong size)
samtools view: writing to "tumor.bam" failed: No space left on device
[E::bgzf_close] File write failed
samtools view: error closing "tumor.bam": -1
/cromwell_root/script: line 25: echo: write error: No space left on device
tee: /cromwell_root/stderr: I/O error

The call is here: https://github.com/wustl-oncology/analysis-wdls/blob/4d3de66e39cb1fbfb10c5e9a5415f0c92df0cdcd/definitions/subworkflows/generate_fda_metrics.wdl#L155-L162

Which calls here: https://github.com/wustl-oncology/analysis-wdls/blob/4d3de66e39cb1fbfb10c5e9a5415f0c92df0cdcd/definitions/tools/cram_to_bam.wdl#L12

Which defines space needs like this: Int space_needed_gb = 10 + round(size([cram, cram_index, reference, reference_index, reference_dict], "GB"))

It does seem like this is likely to be insufficient for larger input files. The space request is based on the size of the CRAM with no multiplier + 10Gb. But the expanded BAM can be much larger than the CRAM. So I think we want some kind of multiplier here.

Going to try this: Int space_needed_gb = 10 + round(size([cram, cram_index, reference, reference_index, reference_dict], "GB") * 2)

[malachig]

Increasing to 2x worked for one dataset but still failed for another. I'm not sure of a reliable way to predict the degree of expansion because it depends on both the size of the BAM but also its complexity. In theory a very large BAM can shrink to almost no size at all if it largely consists of pileups of reads at the same positions. Will try increasing to 3x the size of inputs for now.

[malachig]

This worked.