cutadapt, ipython cluster ncpus exceed error

[zeneofa]

Hi,

When running a subset of the rnaseq pipeline (without the differential expression calling part). I seem to run into a problem at the start when cutadapt is removing adapters. For some reason the ncpus that it appears to use exceed the limit requested of 1. The doc for cutadapt does not mention that it threads, or uses more than one cpu. (I am running this on a PBSPro cluster, using the ipython flag) So don't know if I am missing something.

Here is a link to information on the problem: https://groups.google.com/forum/#!topic/biovalidation/24dX-6C25us

I tried setting the number of cores requested for cutadapt to 2, by adding it to my bcbio_system.yaml. But, the number of cores requested still remains 1. And the problem persists.

Tried using a different data set and I got:

2015-01-28 17:23:58.902 [IPEngineApp] Registering with controller at tcp://127.0.0.1:41229 2015-01-28 17:23:58.974 [IPEngineApp] Starting to monitor the heartbeat signal from the hub every 60010 ms. 2015-01-28 17:23:58.977 [IPEngineApp] Using existing profile dir: u'/lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/project/work/log/ipython' 2015-01-28 17:23:58.979 [IPEngineApp] Completed registration with id 0 =>> PBS: job killed: ncpus 1.57 exceeded limit 1 (sum)

as the error output from on of the bcbio-engine submissions.

more output: -bash-3.2$ tail log/bcbio-nextgen-debug.log [2015-01-28T15:26Z] cnode-7-44: Timing: organize samples [2015-01-28T15:26Z] cnode-7-44: ipython: organize_samples [2015-01-28T15:26Z] cnode-4-69: Using input YAML configuration: /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/project/config/project.yaml [2015-01-28T15:26Z] cnode-4-69: Checking sample YAML configuration: /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/project/config/project.yaml [2015-01-28T15:26Z] cnode-4-69: Testing minimum versions of installed programs [2015-01-28T15:26Z] cnode-7-44: Timing: adapter trimming [2015-01-28T15:26Z] cnode-7-44: ipython: prepare_sample [2015-01-28T15:26Z] cnode-7-44: ipython: trim_sample [2015-01-28T15:26Z] cnode-4-69: Trimming low quality ends and read through adapter sequence from /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/input/UHR_100ng_1_R1.fastq.gz, /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/input/UHR_100ng_1_R2.fastq.gz. [2015-01-28T15:26Z] cnode-4-69: Trimming /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/input/UHR_100ng_1_R1.fastq.gz and /lustre/SCRATCH5/users/pjones/data_files/test_area/bcbio_rnaseq_qc/input/UHR_100ng_1_R2.fastq.gz in paired end mode with cutadapt.

my template yaml:

-bash-3.2$ more template_rnaseq.yaml fc_name: test_pipe details:

• analysis: RNA-QC genome_build: hg_ercc algorithm: trim_reads: read_through adapters: [truseq, ploya] strandedness: [firststrand] upload: dir: ../final

the pipeline (in pipeline/main.py)

class RnaseqQCPipeline(AbstractPipeline): name = "RNASeqQC"

@classmethod
def run(self, config, config_file, parallel, dirs, samples):
    with prun.start(_wres(parallel, ["picard", "AlienTrimmer"]),
                    samples, config, dirs, "trimming") as run_parallel:
        with profile.report("adapter trimming", dirs):
            samples = run_parallel("prepare_sample", samples)
            samples = run_parallel("trim_sample", samples)
    with prun.start(_wres(parallel, ["aligner", "picard"],
                          ensure_mem={"tophat": 8, "tophat2": 8, "star": 40}),
                    samples, config, dirs, "alignment",
                    multiplier=alignprep.parallel_multiplier(samples)) as run_parallel:
        with profile.report("alignment", dirs):
            samples = disambiguate.split(samples)
            samples = run_parallel("process_alignment", samples)
    with prun.start(_wres(parallel, ["picard", "fastqc", "rnaseqc", "kraken"]),
                    samples, config, dirs, "qc") as run_parallel:
        with profile.report("quality control", dirs):
            samples = qcsummary.generate_parallel(samples, run_parallel)
    logger.info("Timing: finished")
    return samples

any advise?

P

[chapmanb]

Piet; Sorry about the problem and missing your earlier message. It looks like you need to swap the resource specification in your custom runner to use cutadapt rather than AlienTrimmer, like:

https://github.com/chapmanb/bcbio-nextgen/blob/02f56ca9d555c2dc9c664d9aa065830f3a0a6321/bcbio/pipeline/main.py#L273

This swapped over with the change in trimmers. That should then respect your core specification and hopefully avoid issues. I don't think cutadapt is multi-threaded so not sure why it is slightly over-running core usage during processing. PBSPro is apparently pretty tight with usage controls so hopefully adjusting your custom code will help.

[zeneofa]

Hi Brad, I changed the script as you suggested, but stil get the 1.59 ncpus. Until I actually run the code as: bcbio_nextgen.py -t ipython -q workq -n 2 -s pbspro (with the core specification in bcbio_system.yaml file as cores: 2 for cutadapt)

Is this the best way to overcome this over-running? ( I am thinking in terms of scale).

Also is there plans to implement the call to cutadapt such that it takes splits of the input fastq files and then processes them (or does it already do this), as this might improve its speed? As cutadapt will most likely not be threaded in future,

https://code.google.com/p/cutadapt/issues/detail?id=44

[zeneofa]

To get the trimming part to complete I had to change the line in trim.py L112 (the return line for the _cutadapt_pe_nosickle method) from:

return first_cmd + ";" + second_cmd + "; rm {tmp_fq1} {tmp_fq2} "

to

return first_cmd + ";" + second_cmd #+ "; rm {tmp_fq1} {tmp_fq2} "

as it complained that the files could not be found, when running the second_cmd option, no idea entirely why though. The trimming then completed without trouble, and the trimmed '.tmp' fastq.gz files were not present in the trimmed folder only the trimmed fastq.gz files.

Testing the rest of the pipeline now.

[chapmanb]

Piet; Glad you were able to get things working with multiple cores. Longer term, I'm confused at all of this -- both cutadapt resource usage and why not removing the temporary files would influence the second command.

I'll have to defer to @roryk on this and possibility of speeding up trimming. I don't know much about cutadapt internals or other choices out there for doing this.

[roryk]

Hi Piet,

Thanks for the updates. I'm confused about why that fix works as well. That is a good idea about splitting up the files to speed it up, I'll work on implementing that for the next version.

[roryk]

I've implemented skewer support as a faster option for trimming, it can get turned on by putting "skewer" under tools_on.