chapmanb
commented
8 years ago Thanks for the report and sorry about the problems. The first issue with BED files looks like there might be empty or truncated lines in an input BED file. I'm not sure what happened there but pushed a fix to the BED file cleanup to try and resolve it. If you update with:
bcbio_nextgen.py upgrade -u developmentIt should hopefully avoid that and work cleanly.
The second issue is Java complaining that it's unable to write to the temporary directory: /lustre/project/work/tx/tmp9PC3iD in this case. My guess is that this is a temporary issue due to shared filesystem weirdness and re-running with work cleanly. If you have consistent issues there it might be worth digging into why you'd have write permission issues there.
Hope this helps get your analyses moving.
pengxiao78
Hi Brad, I found the following new multiple errors by using the same tumor-normal pipeline that I kept running successfully recently. In the slurm file, I used --retries 3 option. Could you please help me to debug them? Thank you!
[2016-01-07T21:58Z] c4401.xxx.edu: Recalibrating ['', '11351tumor'] with GATK [2016-01-07T22:07Z] c4314.xxx.edu: Assigned coverage as 'amplicon' with 6.9% genome coverage and 0.0% offtarget coverage [2016-01-07T22:08Z] c4314.xxx.edu: Uncaught exception occurred Traceback (most recent call last): File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run _do_run(cmd, checks, log_stdout) File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; cat /lustre/project/work/align/1049tumor/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed | grep -v ^track | grep -v ^browser | grep -v ^# | ./bcbio/anaconda/bin/py -x 'bcbio.variation.bedutils.remove_bad(x)' | sort -V -k1,1 -k2,2n > /lustre/project/work/bedprep/tx/tmp5Eu_W3/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed cat: /lustre/project/work/align/1049tumor/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed: Input/output error Traceback (most recent call last): File "", line 1, in
bcbio.variation.bedutils.remove_bad(x)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/bedutils.py", line 70, in remove_bad
if int(parts[2]) > int(parts[1]):
IndexError: list index out of range
' returned non-zero exit status 1
[2016-01-07T22:08Z] c4314.xxx.edu: Unexpected error
Traceback (most recent call last):
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 44, in _setup_logging
yield config
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 133, in postprocess_alignment
return ipython.zip_args(apply(sample.postprocess_alignment, _args))
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/sample.py", line 164, in postprocess_alignment
data = bedutils.clean_inputs(data)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/bedutils.py", line 106, in clean_inputs
clean_vr = clean_file(utils.get_in(data, ("config", "algorithm", "variant_regions")), data)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/bedutils.py", line 49, in cleanfile
do.run(cmd.format(*locals()), "Prepare cleaned BED file", data)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
_do_run(cmd, checks, log_stdout)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
raise subprocess.CalledProcessError(exitcode, error_msg)
CalledProcessError: Command 'set -o pipefail; cat /lustre/project/work/align/1049tumor/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed | grep -v ^track | grep -v ^browser | grep -v ^# | ./bcbio/anaconda/bin/py -x 'bcbio.variation.bedutils.remove_bad(x)' | sort -V -k1,1 -k2,2n > /lustre/project/work/bedprep/tx/tmp5Eu_W3/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed
cat: /lustre/project/work/align/1049tumor/8_2016-01-05_project_1st-merged-sort-callable-callableblocks.bed: Input/output error
Traceback (most recent call last):
File "", line 1, in
bcbio.variation.bedutils.remove_bad(x)
File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/bedutils.py", line 70, in remove_bad
if int(parts[2]) > int(parts[1]):
IndexError: list index out of range
' returned non-zero exit status 1
[2016-01-07T22:14Z] c4305.xxx.edu: Assigned coverage as 'amplicon' with 20.7% genome coverage and 0.0% offtarget coverage
[2016-01-07T22:15Z] c4305.xxx.edu: Recalibrating ['', '5001normal'] with GATK
[2016-01-07T22:24Z] c4415.xxx.edu: Assigned coverage as 'amplicon' with 6.9% genome coverage and 0.0% offtarget coverage
[2016-01-07T22:24Z] c4415.xxx.edu: Recalibrating ['', '1049tumor'] with GATK
[2016-01-07T23:21Z] c0905.xxx.edu: ipython: combine_sample_regions
[2016-01-07T23:44Z] c4308.xxx.edu: Identified 170 parallel analysis blocks
Block sizes:
min: 4262
5%: 27977.65
25%: 163650.5
median: 31018084.5
75%: 31019594.25
95%: 31024801.8
99%: 39108842.99
max: 58819774
Between block sizes:
min: 259
5%: 273.2
25%: 329.5
median: 423.0
75%: 676.5
95%: 1388.3
99%: 4115.46
max: 14211
[2016-01-07T23:44Z] c0905.xxx.edu: Timing: structural variation initial [2016-01-07T23:44Z] c0905.xxx.edu: ipython: detect_sv [2016-01-08T18:17Z] c0905.xxx.edu: Timing: alignment post-processing [2016-01-08T18:17Z] c0905.xxx.edu: ipython: piped_bamprep [2016-01-08T18:53Z] c4415.xxx.edu: Uncaught exception occurred Traceback (most recent call last): File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run _do_run(cmd, checks, log_stdout) File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; ./bcbio/bin/sambamba index -t 1 /lustre/project/work/bamprep/10132normal/4/tx/tmpYLFbC6/5_2016-01-05_project_1st-merged-sort-4_0_31019430-prep-prealign.bam sambamba-index: Unable to write to stream ' returned non-zero exit status 1 [2016-01-08T18:53Z] c4415.xxx.edu: Uncaught exception occurred Traceback (most recent call last): File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run _do_run(cmd, checks, log_stdout) File "./bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; java -Xms166m -Xmx1166m -XX:+UseSerialGC -Djava.io.tmpdir=/lustre/project/work/tx/tmp9PC3iD -jar ./bcbio/toolplus/gatk/3.4-46-gbc02625/GenomeAnalysisTK.jar -T PrintReads -L 7:155102739-159138663 -R ./bcbio/genomes/Hsapiens/GRCh37/seq/GRCh37.fa -I /lustre/project/work/align/10132normal/5_2016-01-05_project_1st-merged-sort.bam -BQSR /lustre/project/work/align/10132normal/5_2016-01-05_project_1st-merged-sort.grp -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment -o /lustre/project/work/bamprep/10132normal/7/tx/tmp2c1e4Y/5_2016-01-05_project_1st-merged-sort-7_155102738_159138663-prep-prealign.bam INFO 12:53:07,427 HelpFormatter - --------------------------------------------------------------------------------- INFO 12:53:07,429 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.4-46-gbc02625, Compiled 2015/07/09 17:38:12 INFO 12:53:07,429 HelpFormatter - Copyright (c) 2010 The Broad Institute INFO 12:53:07,429 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk INFO 12:53:07,433 HelpFormatter - Program Args: -T PrintReads -L 7:155102739-159138663 -R ./bcbio/genomes/Hsapiens/GRCh37/seq/GRCh37.fa -I /lustre/project/work/align/10132normal/5_2016-01-05_cowan_1st-merged-sort.bam -BQSR /lustre/project/work/align/10132normal/5_2016-01-05_project_1st-merged-sort.grp -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment -o /lustre/project/work/bamprep/10132normal/7/tx/tmp2c1e4Y/5_2016-01-05_project_1st-merged-sort-7_155102738_159138663-prep-prealign.bam INFO 12:53:07,439 HelpFormatter - Executing as xxx@c4415.xxx.edu on Linux 2.6.32-504.23.4.el6.x86_64 amd64; OpenJDK 64-Bit Server VM 1.7.0_85-mockbuild_2015_07_15_13_21-b00. INFO 12:53:07,440 HelpFormatter - Date/Time: 2016/01/08 12:53:07 INFO 12:53:07,440 HelpFormatter - --------------------------------------------------------------------------------- INFO 12:53:07,440 HelpFormatter - --------------------------------------------------------------------------------- INFO 12:53:07,566 GenomeAnalysisEngine - Strictness is SILENT INFO 12:53:08,465 ContextCovariate - Context sizes: base substitution model 2, indel substitution model 3 INFO 12:53:08,528 GenomeAnalysisEngine - Downsampling Settings: No downsampling INFO 12:53:08,537 SAMDataSource$SAMReaders - Initializing SAMRecords in serial INFO 12:53:08,578 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.04 INFO 12:53:08,596 IntervalUtils - Processing 4035925 bp from intervals INFO 12:53:08,666 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files INFO 12:53:15,227 GATKRunReport - Uploaded run statistics report to AWS S3
ERROR ------------------------------------------------------------------------------------------
ERROR A USER ERROR has occurred (version 3.4-46-gbc02625):
ERROR
ERROR This means that one or more arguments or inputs in your command are incorrect.
ERROR The error message below tells you what is the problem.
ERROR
ERROR If the problem is an invalid argument, please check the online documentation guide
ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
ERROR
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions http://www.broadinstitute.org/gatk
ERROR
ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
ERROR
ERROR MESSAGE: Unable to create a temporary BAM schedule file. Please make sure Java can write to the default temp directory or use -Djava.io.tmpdir= to instruct it to use a different temp directory instead.
ERROR ------------------------------------------------------------------------------------------
' returned non-zero exit status 1