Closed alexg9010 closed 1 year ago
Here is some collected from the recent run with Bismark --multicore 3
(took ~ 4days ):
input:
temp files during mapping:
--> up to ~ 660G per Sample
Considering we would only run one instance of Bismark instead of three we could probably reduce tempfiles ton 220G per sample. I do not know what the loss in runtime would be, but I assume it would be very high considering we have to write fewer times to the disk.
I will test my idea by using the args
in the settings file and then report wether this will help.
RP6_bismark_pe_mapping.multicore3.log RP6_bismark_pe_mapping.parallel8.log
--multicore 3
==============
bismark instances: 3
bismark threads: 3 * 1 = 3
bowtie instances: 3 * 2 = 6
bowtie threads: 6 * 1 = 6
samtools threads: 6 * 1 = 6
gzip threads: 6 * 1 = 6
total threads: 3 + 6 + 6 + 6 = 21
runtime: 4d 11h 32m 19s
approx. ressources:
------------
hulk memory usage during run (28 samples plus other users) : ~ 700 GB
disk: 14 TB/d in 1 h = 14/24 = ~ 0.58 TB = 600 GB
-p 8
=============
bismark instances: 1
bismark threads: 1
bowtie instances: 2 = 2
bowtie threads: 2 * 8 = 16
samtools threads: 2 * 1 = 2
gzip threads: 2 * 1 = 2
total threads: 1 + 16 + 2 + 2 = 21
runtime: 1d 6h 2m 42s
appro. ressources:
------------
hulk memory usage during run (1 sample plus other users): ~ 200 GB
disk: 14 TB/d in 1 h = 14/24 = ~ 0.58 TB = 600 GB
Any update on this issue? I also met the issue of slow run of bismark on one sample, and each bowtie2 instance can use 30GB memory. Need some clues to fix it.
Hi @fortune9,
As you can see from my previous comment, in our case it seemed that using fewer parallel instances (--multicore
3 -> 1) and instead dedicating more threads to bowtie2 (-p
1 -> 8 ) lead to a 3.5X increase in processing speed without a major increase in memory usage.
In case you are using the pigx-bsseq pipeline, this would correspond to changing the bismark arguments in the tools section from the default (--multicore 3 -N 0 -L 20
):
tools:
bismark:
args: " -N 0 -L 20 "
cores: 3
to this (--multicore 1 -p 8 -N 0 -L 20
):
tools:
bismark:
args: "-p 8 -N 0 -L 20 "
cores: 1
I recently run the pipeline on a large dataset with 28 samples and during this I made some observations:
When using N multiple cores (
cores
set in settings file,--multicore
set in bismark call), we are producing parallel instances of Bismark, i.e. Bismark will take the original (gzipped) fastq file, split it into subsets of N uncompressed fastq files and then run Bismark workflow on each subset, where 2/4 instances (directional/non-directional-mode) of Bowtie2 are run, each with 1 thread. This way we might be using less resources (cores/mem) per bowtie2 instance, but are producing a lot IO on the disk, which in general is slower than keeping stuff in memory.See multicore setting in Bismark: https://github.com/FelixKrueger/Bismark/blob/master/docs/options/alignment.md#alignment
What we could do instead / in addition is to use adjust the number of threads per bowtie2 instance using the
-p
argument to Bismark and decrease the number of parallel Bismark instances (--multicore
), which would increase the resource usage, but should decrease the IO load.https://github.com/FelixKrueger/Bismark/blob/master/docs/options/alignment.md#parallelization-options