[REQUEST] Right size obvious tasks to alleviate resource exhaustion from CPU quota

Feature Request

We are reaching our CPU limit for our production Terra workspace. This is negatively impacting our pipeline and preventing us from reliably meeting dashboard updates.

This issue is concerned primarily with the create_version_capture_file task:

https://github.com/CDPHE-bioinformatics/CDPHE-SARS-CoV-2/blob/c988ea7924bdd323adbd099ad38fb7ca60d18d5d/workflows/SC2_illumina_pe_assembly.wdl#L597

https://github.com/CDPHE-bioinformatics/CDPHE-SARS-CoV-2/blob/c988ea7924bdd323adbd099ad38fb7ca60d18d5d/workflows/SC2_ont_assembly.wdl#L537

Solution

To fix this we should review our container runtime settings in our WDLs. There are a few places where they need to be fixed:

reduce CPU, memory, and disks to be appropriate
remove unused or invalid runtime settings (e.g. dx_instance_type)
consider updating docker containers if needed

Other possible changes

We should also consider reducing the transfer task to use fewer CPU. While this will increase the transfer task time, it'll free up CPUs so more tasks can be spawned at once. The trade off makes sense to me since spawning other tasks that have long runtimes is better than trying to get through file transfers that don't take that long to begin with.

https://github.com/CDPHE-bioinformatics/CDPHE-SARS-CoV-2/blob/c988ea7924bdd323adbd099ad38fb7ca60d18d5d/workflows/SC2_illumina_pe_assembly.wdl#L667

https://github.com/CDPHE-bioinformatics/CDPHE-SARS-CoV-2/blob/c988ea7924bdd323adbd099ad38fb7ca60d18d5d/workflows/SC2_ont_assembly.wdl#L592

Also, if a WDL command block runs serially, we might not see a benefit to multiple CPUs in the transfer tasks anyway.

Downstream effects

All other tasks should be examined as well, but those should be their own issues since we can accomplish a bit with this low hanging fruit.

CDPHE-bioinformatics / CDPHE-SARS-CoV-2