Cumulus RDPC workflows benchmark

icgc-argo / workflow-roadmap

Roadmap and management for genomic data processing

GNU Affero General Public License v3.0

1 stars 0 forks source link

Cumulus RDPC workflows benchmark #408

Closed lindaxiang closed 5 months ago

lindaxiang commented 10 months ago

We're using P1000-US DNA-Seq alignment jobs as dataset for benchmark. The purposes are to determine:

Proper cpu/memory setting for running each DNA-Sequencing task, especially bwa-mem module
Proper number of bwa-mem tasks per node
Max number of jobs per workDir

lindaxiang commented 10 months ago

Cluster setting between Jan 2-14, 2024 1 job/workDir 14 nodes with 64CPU/128GB bwa-mem request: 18cpu/56GB

justincorrigible commented 8 months ago

To confirm whether we still need this ticket

edsu7 commented 5 months ago

Second update see: https://docs.google.com/presentation/d/1QThEXonaT9zcEjs8NfNqNuPy6qsoJnDeavTgeWYMO8U/edit#slide=id.g2e1d6ca196e_0_23

Conclusion: Not entirely comparable as workflows were mainly mutect2 and less IO heavy compared to alignments, therefore may not reflect refactor changes.

Action items:

open new ticket to explore once more alignment workflows are available