Bismark not being run in multicore mode

[colindaven]

Hi,

I am trying to get the docker pipeline optimised and running quickly.

I assume the docker.config file is causing my CPU settings to be reset to 1. Where can I override this ? Do I need to just avoid importing the docker.config ? I already tried setting cpus=48 in the docker.config without success, and increasing limits in the base.config

conf/docker.config

/*
 * NOTE: Not suitable for production use, assumes
 * compute limits of only 16GB memory and 1 CPU core.
 */

params {
  igenomes_base = 's3://ngi-igenomes/igenomes/'
  cpus = 48
}

Thanks, otherwise the pipeline is looking excellent. Colin

[ewels]

Hi Colin,

Apologies, that comment about being not suitable for production use is out of date and not true any more. I'll remove that now.

When you use -profile docker for a pipeline run, Nextflow uses the following config files: https://github.com/SciLifeLab/NGI-MethylSeq/blob/a3398cd1fcad81665b4d20d4893fc3ae25fdcd30/nextflow.config#L41-L45

The docker config file you paste doesn't contain any settings for resource requirements. Those are set in base.config.

By default, jobs do use 1 CPU. However, most tasks in the pipeline override this default to make optimal use of resources. For example, Bismark is configured to use 8 cpus and 64GB memory: https://github.com/SciLifeLab/NGI-MethylSeq/blob/a3398cd1fcad81665b4d20d4893fc3ae25fdcd30/conf/base.config#L37-L41

Which tasks would you like to use 48 cpus for? In my experience, going above around 8 cpu parallelisation for bismark alignment you start to get diminishing returns because of the high IO required. Note that Nextflow should automatically manage the total requirements, so if your system has 48 cpus and you're running in local mode (the default for the docker profile) then there should be up to six alignment tasks running in parallel (six different samples).

Phil

[colindaven]

Hi Phil,

Thanks for the very clear response and clearing up the docker comment.

Yeah, of course you're right about the 48 cores, was just trying to get any response out of it. I have rolled back all changes by rebasing and repulling but still don't see more than one core being used, especially in the alignment phase. Multicore adapter trimming would be nice too though.

I will try recloning the whole thing and trying a different server as well.

This is the file (SLURM sbatch file, here running with bash to capture output directly)

public_bisulfite> less run_scilife_bisulfite_SLURM.sh 
#!/bin/bash

# set partition - for IO, star
#SBATCH -p long

# set run on x MB node only
#SBATCH --mem 128000

# set run on bigmem node only
#SBATCH --cpus-per-task 16

# share node
#SBATCH --exclusive

# set max wallclock time
#SBATCH --time=87:00:00

# set name of job
#SBATCH --job-name=scil_bisulf

# Add miniconda3 to PATH
#PATH=/mnt/ngsnfs/vmstore/tuem/tools/miniconda3/bin:$PATH
#source /mnt/ngsnfs/globalenv/globalenv

# Activate env on cluster node
#source activate PPKC_env

#PE reads _sm2 - no nextflow.config selected
nextflow run /mnt/ngsnfs/tools/NGI-MethylSeq/bismark.nf --bismark_index '/lager2/rcug/seqres/HS/bismark/GRCh38_no_alt/' --reads 'SRR4235786_R{1,2}_sm2.fastq' -p
rofile docker

$ bash run_scilife_bisulfite_SLURM.sh 

Picked up _JAVA_OPTIONS: -Dhttp.proxyHost=172.24.2.50 -Dhttp.proxyPort=8080 -Dhttps.proxyHost=172.24.2.50 -Dhttps.proxyPort=8080
Picked up _JAVA_OPTIONS: -Dhttp.proxyHost=172.24.2.50 -Dhttp.proxyPort=8080 -Dhttps.proxyHost=172.24.2.50 -Dhttps.proxyPort=8080
N E X T F L O W  ~  version 0.26.2
Launching `/mnt/ngsnfs/tools/NGI-MethylSeq/bismark.nf` [zen_jang] - revision: 79c177e468
WARN: The config file defines settings for an unknown process: bwamem_align -- Did you mean: bismark_align?
WARN: The config file defines settings for an unknown process: samtools_sort
WARN: The config file defines settings for an unknown process: markDuplicates
WARN: The config file defines settings for an unknown process: methyldackel
==================================================
 NGI-MethylSeq : Bisulfite-Seq Best Practice v0.4dev
==================================================
Run Name          : zen_jang
Reads             : SRR4235786_R{1,2}_sm2.fastq
Data Type         : Paired-End
Genome            : false
Bismark Index     : /lager2/rcug/seqres/HS/bismark/GRCh38_no_alt/
Trim R1           : 0
Trim R2           : 0
Trim 3' R1        : 0
Trim 3' R2        : 0
Deduplication     : Yes
Directional Mode  : Yes
All C Contexts    : No
Save Reference    : No
Save Trimmed      : No
Save Unmapped     : No
Save Intermeds    : No
Max Memory        : 128 GB
Max CPUs          : 16
Max Time          : 10d
Output dir        : ./results
Working dir       : /lager2/rcug/2018/public_bisulfite/work
Container         : 
Current home      : /home/rcug
Current user      : rcug
Current path      : /lager2/rcug/2018/public_bisulfite
Script dir        : /mnt/ngsnfs/tools/NGI-MethylSeq
Config Profile    : docker
=========================================
[warm up] executor > local
[56/abca15] Submitted process > fastqc (SRR4235786_R)
[ac/4dfb2a] Submitted process > get_software_versions
[d4/6640e6] Submitted process > trim_galore (SRR4235786_R)

[ewels]

Multicore adapter trimming would be nice too though.

Yes, though when I wrote this cutadapt (the trimming tool used by Trim Galore!) didn't support multi-core parallelisation. A new version came out fairly recently which does support this though, so I should probably check whether TrimGalore! can pass that on.. (@FelixKrueger?).

but still don't see more than one core being used, especially in the alignment phase.

Hmm, so looking at the pipeline script it looks like I haven't specified --multicore: https://github.com/SciLifeLab/NGI-MethylSeq/blob/9cb54ba4d1dfee7a2f679598582aa5246f8a2e5b/bismark.nf#L324-L328

So I guess that even if Nextflow is assigning 16 cpus for you, Bismark probably isn't using them for the alignment. I should update the pipeline to get it to use the cpus it's being given..

Phil

[ewels]

Ok, changes done in https://github.com/ewels/NGI-MethylSeq/commit/ecc535cf632f66cb8fb331b8c2fbeb50f7f3f611

Will run a few tests to make sure that everything works as expected and then do a pull-request to this repository.

[colindaven]

Excellent, cheers. Unfortunately my test of ewels/NGI-MethylSeq@ecc535c gave me an error:

Launching `/mnt/ngsnfs/tools/test/NGI-MethylSeq/bismark.nf` [small_northcutt] - revision: 86fdcc6652
ERROR ~ unexpected token: / @ line 326, column 46.
    Math.max(1, task.cpus as int / cpu_per_
                                 ^

1 error

 -- Check '.nextflow.log' file for details

Hope that helps ....

[ewels]

NB: Multi-core trimming isn't supported by TrimGalore! yet, so can't implement that sorry.

[ewels]

Yes, I pushed to my fork before I had run any tests. I've fixed that error now and have just pushed a fix. I'll run some more tests and then do a pull-request to merge into this repository when it's ready.

Phil

[ewels]

Hi @colindaven,

Apologies, I haven't had time to run any additional tests yet. I'll create a pull request anyway though. If you get a chance to play with my fork as above then that would be brilliant 👍

Phil

[ewels]

See #42

[colindaven]

Hey @ewels

I have been playing with the fork. It's looking pretty good on the multicore front (am getting 8 cores I believe) but I am just starting out with the pipeline so don't know if everything is running through ok. I also haven't used Bismark for ~2 years now so might be missing a bit.

Looks like I am getting dedup bams and methylation in multiple formats plus pipeline reports, so I am pretty happy so far. Qualimap, MultiQC and TrimGalore also seem complete. I'll let you know if I find any omissions, but I don't have loads of files to test at the moment.

Cheers for this, it's pretty close to plug and play, the only difficulty has been optimising for the large input files.

[ewels]

Ok brilliant, thanks! The main thing I'd like to do is check the Nextflow trace report to see what percentage of requested CPUs and Memory is being used by the bismark align and methXtract steps. If the code is working well, it should be fairly efficient in that plot. I've also planned to but never done more involved resource testing too (our cluster has a bunch of tools to profile cpu and memory usage over the lifetime of a cluster job).

Note that I'm recently working on a new project to "un-brand" our Nextflow pipelines to encourage others to use and contribute. So this pipeline will be renamed and moved to nf-core/MethylSeq soon instead of SciLifeLab/NGI-MethylSeq. See https://nf-core.github.io/ for more information.

I'm thinking that it would be a good time to refactor the pipeline a bit too - for example, merging the two aligner scripts into one (instead just have --aligner bwameth to override the default of bismark). Let me know if you come up with any other ideas.

Phil

[ewels]

Closing this now - let me know if we should come back to the usage requirements and I'll reopen.