walaj
commented
6 years ago Hi Brad,
I've thought about this before, but don't really see a good solution. I just used valgrind to profile, and as expected the majority of the compute is just reading and writing the reads. In so far as even samtools doesn't do multi-core reading/writing, I think implementing this efficiently for VariantBam without messing with the order of the reads or having to merge would be very difficult.
Quick benchmarks:
samtools view BAM to SAM for 1m reads: 5.3s
variant BAM to SAM (no filter) for 1m reads: 6.2s
variant -m 100 BAM to SAM (max coverage 100) for 1m reads: 8.6s
So it looks like there's a small penalty to do the coverage calcs / downsampling, but the bottleneck remains the reading/writing steps. One caveat is that SeqAn library has a way to multicore reading/writing, since most of the BAM read/write compute is in the BGZF compression/deflation rather than I/O.
chapmanb
ohofmann
brainstorm
Jeremiah; Now that we have VariantBam maximum coverage downsampling in bcbio (thank you again), we've been working on doing benchmarking runs with and without downsampling. Because we need to run it on sorted BAMs it ends up running essentially on it's own as the streaming output comes from bamsormadup. Since it's single threaded, this adds a noticeable amount of time to processing whole genome BAMs (where the downsampling is most useful). I wanted to pick your brain to see if there were options to run multicore or otherwise parallelize maximum coverage downsampling to improve runtimes. My only thought was to try and parallelize by region but then we start running into IO and merging bottlenecks which may not provide much of an improvement. Thanks for any suggestions or tips on speeding up the downsampling.