search

broadinstitute / gatk-sv

A structural variation pipeline for short-read sequencing
BSD 3-Clause "New" or "Revised" License
170 stars 70 forks source link

Modify WDL to Use N2 Machine Types for Enhanced Memory Allocation #621

Closed shadizaheri closed 5 months ago

shadizaheri commented 10 months ago

Motivation While executing SVConcordance, memory allocation issues have arisen specifically for chromosomes 9, 17, and X. Despite efforts to allocate more memory to the SV_concordance task, limitations from Google Cloud Platform (GCP) have capped the maximum allocatable memory to 300GB. This constraint has hindered the successful processing of these chromosomes.

Proposed Solution The proposed solution involves modifying the Workflow Description Language (WDL) script to utilize N2 machine types. By specifying the cpuPlatform: "Intel Cascade Lake" within the runtime block of the WDL, it is possible to access larger machines, thereby facilitating greater memory allocation.

Implementation Details The following changes will be made in the WDL script:

runtime {
    ...
    cpuPlatform: "Intel Cascade Lake"
    ...
}

This modification will instruct the workflow to utilize N2 machine types that are capable of supporting larger memory requirements, essential for processing the specified chromosomes without encountering memory constraints.

Anticipated Outcome The implementation of N2 machine types is expected to resolve the memory allocation issues, allowing SVConcordance to run efficiently for chromosomes 9, 17, X, and other chromosomes without hitting the memory cap imposed by GCP.

mwalker174 commented 5 months ago

Closing this. I don't believe we were able to secure higher capacity machines this way and we've already fixed the memory issue in #623.