search

heathsc / gemBS

gemBS is a bioinformatics pipeline designed for high throughput analysis of DNA methylation from Whole Genome Bisulfite Sequencing data (WGBS).
GNU General Public License v3.0
32 stars 21 forks source link

Scattering gemBS alignment step across a cluster #84

Open gilhornung opened 3 years ago

gilhornung commented 3 years ago

Hello, We are using Cromwell on a Google Cloud Platform to analyze sequencing data. We found that, at least for bwa and WGS, scattering the reads in small chunks across several nodes with few cpus is cheaper than running the entire sample on one node with many cpus, when using preemptible instances. From the documentation I saw that for gemBS the recommendation is to run all steps on one machine with multiple cpus for parallelization. My question is whether it is possible to perform the map step of gemBS across many independent instances, and then merge the resulting bams using gemBS. Thank you, Gil

heathsc commented 3 years ago

In principle yes (we don't use cromwell, but we have a workflow based on Slurm which splits the run across a cluster without any major problems). If there is a shared file system for all nodes then it should work pretty much without modification, otherwise you'll have to copy the bams for each chunk into the same directory and use the --no-db option to gemBS for the merge step.

Simon

On Wed, Jul 7, 2021 at 2:31 PM gilhornung @.***> wrote:

Hello, We are using Cromwell on a Google Cloud Platform to analyze sequencing data. We found that, at least for bwa and WGS, scattering the reads in small chunks across several nodes with few cpus is cheaper than running the entire sample on one node with many cpus, when using preemptible instances https://cloud.google.com/compute/docs/instances/preemptible. From the documentation I saw that for gemBS the recommendation is to run all steps on one machine with multiple cpus for parallelization. My question is whether it is possible to perform the map step of gemBS across many independent instances, and then merge the resulting bams using gemBS. Thank you, Gil

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/84, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY465Z7NQFKJU4QVSD3JM3TWRCKNANCNFSM476SG54Q .