search

Illumina / manta

Structural variant and indel caller for mapped sequencing data
GNU General Public License v3.0
404 stars 154 forks source link

Another " Failed to complete command task: 'generateCandidateSV_0000' " #217

Open Fred-07 opened 4 years ago

Fred-07 commented 4 years ago

Manta is launched as an LSF command requesting 8 cores. It fails on particular data from 1000 genomes (HG002, HG003, HG004), but works on internal samples. All samples (1000 genomes or internal) were processed the same, starting from the .fastq reads.

Fred-07 commented 4 years ago

[2020-03-04T16:18:07.428228Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] Launching command task: 'generateCandidateSV_0000' from master workflow [2020-03-04T16:18:07.449977Z] [cpt01.cluster.uni.ch] [34169_1] [TaskRunner:generateCandidateSV_0000] Task initiated on local node [2020-03-04T16:31:44.504454Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] Failed to complete command task: 'generateCandidateSV_0000' laun ched from master workflow, error code: 1, command: '/scratch/Genomes/MainUser/Progs/Manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --t hreads 8 --align-stats /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/alignmentStats.xml --graph-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate- sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /scratch/Genomes/dbs/genome/GRCh37.p13/genome_PAR_masked.fasta --candidate-o utput-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 -- min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/ch romDepth.txt --edge-runtime-log /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/edgeRuntimeLog.0000.txt --ed ge-stats-log /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/results/stats/svCandidateGenerationStats.xml --edge-stats-report /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/results/stats/svCandidateGenerationStats.tsv --evidence-bam-stub /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/evidence_0000 --align-file /data/Genomes/MainUser/Genomes_serie0_Fall201 9/1KG_HG_002_sorted_duplicates_removed.bam --output-contigs' [2020-03-04T16:31:44.541930Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Error Message: [2020-03-04T16:31:44.544660Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Anomalous task wrapper stderr output. Wrapper signal file: '/data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/pyflow.data/logs/tmp/taskWrapperLogs/000/311/ pyflowTaskWrapper.signal.txt' [2020-03-04T16:31:44.549632Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] Logging 5 line(s) of task wrapper log output below: [2020-03-04T16:31:44.552874Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:18:07.504467Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] wrapperStart [2020-03-04T16:31:44.555877Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:18:07.618163Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStart [2020-03-04T16:31:44.558440Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:31:44.239821Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskExitCode -11 [2020-03-04T16:31:44.562454Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:31:44.482209Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStderrTail 1 [2020-03-04T16:31:44.565176Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] Last 0 stderr lines from task (of 0 total lines): [2020-03-04T16:31:44.568022Z] [cpt01.cluster.uni.ch] [34169_1] [TaskManager] [ERROR] Shutting down task submission. Waiting for remaining tasks to complete. [2020-03-04T16:31:55.300967Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] Workflow terminated due to the following task errors: [2020-03-04T16:31:55.345519Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] Failed to complete command task: 'generateCandidateSV_0000' launched from master workflow, error code: 1, command: '/scratch/Genomes/MainUser/Progs/Manta/manta-1.6.0.centos6_x86_64/libexec/GenerateSVCandidates --threads 8 --align-stats /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/alignmentStats.xml --graph-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svLocusGraph.bin --bin-index 0 --bin-count 1 --max-edge-count 10 --min-candidate-sv-size 8 --min-candidate-spanning-count 3 --min-scored-sv-size 50 --ref /scratch/Genomes/dbs/genome/GRCh37.p13/genome_PAR_masked.fasta --candidate-output-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/candidateSV.0000.vcf --diploid-output-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/diploidSV.0000.vcf --min-qual-score 10 --min-pass-qual-score 20 --min-pass-gt-score 15 --enable-remote-read-retrieval --chrom-depth /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/chromDepth.txt --edge-runtime-log /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/edgeRuntimeLog.0000.txt --edge-stats-log /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/results/stats/svCandidateGenerationStats.xml --edge-stats-report /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/results/stats/svCandidateGenerationStats.tsv --evidence-bam-stub /data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/svHyGen/evidence_0000 --align-file /data/Genomes/MainUser/Genomes_serie0_Fall2019/1KG_HG_002_sorted_duplicates_removed.bam --output-contigs' [2020-03-04T16:31:55.357486Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Error Message: [2020-03-04T16:31:55.369436Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Anomalous task wrapper stderr output. Wrapper signal file: '/data/Genomes/MainUser/Genomes_serie0_Fall2019/Manta/RunDir_1KG_HG_002/workspace/pyflow.data/logs/tmp/taskWrapperLogs/000/311/pyflowTaskWrapper.signal.txt' [2020-03-04T16:31:55.381437Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] Logging 5 line(s) of task wrapper log output below: [2020-03-04T16:31:55.393440Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:18:07.504467Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] wrapperStart [2020-03-04T16:31:55.405437Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:18:07.618163Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStart [2020-03-04T16:31:55.417440Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:31:44.239821Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskExitCode -11 [2020-03-04T16:31:55.429443Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] [2020-03-04T16:31:44.482209Z] [cpt01.cluster.uni.ch] [34169_1] [pyflowTaskWrapper:generateCandidateSV_0000] [wrapperSignal] taskStderrTail 1 [2020-03-04T16:31:55.439492Z] [cpt01.cluster.uni.ch] [34169_1] [WorkflowRunner] [ERROR] [generateCandidateSV_0000] [taskWrapper-stderr] Last 0 stderr lines from task (of 0 total lines):

Fred-07 commented 4 years ago

The content of the 'result' directory where an anomalous file is created :

RunDir_1KG_HG_002/: total 1120142 -rw------- 1 user diag-dpt 1261289472 Mar 4 17:31 core.51471 -rw-rw-r-- 1 user diag-dpt 220531 Mar 4 17:31 LOG-1 drwxrwxr-x 5 user diag-dpt 5 Mar 4 16:45 results -rwxr-xr-x 1 user diag-dpt 7045 Mar 4 16:45 runWorkflow.py -rw-rw-r-- 1 user diag-dpt 4837 Mar 4 16:45 runWorkflow.py.config.pickle -rw-rw-r-- 1 user diag-dpt 7850 Mar 4 17:31 workflow.error.log.txt -rw-rw-r-- 1 user diag-dpt 2 Mar 4 17:31 workflow.exitcode.txt -rw-rw-r-- 1 user diag-dpt 0 Mar 4 16:55 workflow.warning.log.txt drwxrwxr-x 4 user diag-dpt 7 Mar 4 17:18 workspace

amizeranschi commented 4 years ago

@Fred-07

I'm running into a similar issue: https://github.com/bcbio/bcbio-nextgen/issues/3268

Could this be due to some process using up too much memory and getting killed by the scheduler? An older issue report seems to suggest this: https://github.com/Illumina/pyflow/issues/2