Add auto-retry feature into Sanger variant calling step whenever it is `out of memory` issue

[lindaxiang]

According to the investigation of OOM(https://github.com/icgc-argo/workflow-roadmap/issues/136), we will implement the re-try mechanism for Sanger wgs variant calling step as what Alex suggested.

• nextflow resource: https://www.nextflow.io/docs/latest/process.html#dynamic-computing-resources

tool in scope:

• [ ] sanger variant caller: https://github.com/icgc-argo/variant-calling-tools/blob/main/tools/sanger-wgs-variant-caller/sanger-wgs-variant-caller.nf

workflow in scope

• [ ] sanger-wgs-variant-calling

Note: We will leave sanger wxs calling as it is for now as we did not have OOM issue for wxs data at this time. We can implement similar retry mechanism if it happens to wxs as well.

[lindaxiang]

Related to the investigation here: #136

process sangerWgsVariantCaller { cpus {task.attempt > 1 ? params.cpus - (parms.cpu_backoff * task.attempt) : params.cpus}

errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' } maxRetries params.max_retries }

[puneet-oicr]

Low priority