Closed YaungLiu closed 3 years ago
roryk
commented
3 years ago Hi @Hmllycat,
It's tough to debug this without seeing the whole error-- this is just telling us the command failed, but not why. Could you post more of the log, or the entire log, so that we can take a look? I'm guessing the same issue is causing both problems you posted.
YaungLiu
commented
3 years ago Hi @roryk, thank you for your suggestion, here is the log file, it's quite long so I cannot post all. (If you wanna have a look on the whole file, here is the link
bcbio-nextgen.log
)
""" [2021-01-22T12:35Z] cn513-16-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:35Z] cn514-12-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:35Z] cn603-22-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:35Z] cn509-28-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:36Z] cn506-21-l: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn506-18-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn512-13-l: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn513-13-l: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn514-08-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn514-07-l: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:37Z] cn506-04-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:38Z] cn513-19-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T12:38Z] cn513-15-r: Aligning lane M-19-5931_LS1P0034-V01-LIV1_i7012-i503 with bwa aligner [2021-01-22T13:08Z] cn513-10-r: ipython: delayed_bam_merge [2021-01-22T13:21Z] cn514-12-r: Uncaught exception occurred Traceback (most recent call last): File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) subprocess.CalledProcessError: Command 'set -o pipefail; bamcat level=0 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam-bammerge `cat /encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.list` | bamsormadup threads=16 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam-bamsormaduptmp indexfilename=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam.bai > /encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsv__dfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam [V] 179909860 09:31:05142000 MemUsage(size=4108.39,rss=1993.31,peak=4127.41) AutoArrayMemUsage(memusage=3254.93,peakmemusage=3383.28,maxmem=1.75922e+13) [V] 180355072 [V] 180620318 09:33:32062099 MemUsage(size=4108.39,rss=1993.31,peak=4127.41) AutoArrayMemUsage(memusage=3254.93,peakmemusage=3383.28,maxmem=1.75922e+13) [V] 181403648 [V] 182044596 09:37:80242799 MemUsage(size=4108.39,rss=1993.31,peak=4127.41) AutoArrayMemUsage(memusage=3254.93,peakmemusage=3383.28,maxmem=1.75922e+13) [V] 182452224 [V] 182756277 09:39:51234599 MemUsage(size=4108.39,rss=1993.32,peak=4127.41) AutoArrayMemUsage(memusage=3254.93,peakmemusage=3383.28,maxmem=1.75922e+13) """
"""
[V] 220379477 12:05:81200899 MemUsage(size=6785.15,rss=5833.73,peak=6895.24) AutoArrayMemUsage(memusage=5702.71,peakmemusage=5719.57,maxmem=1.75922e+13)
terminate called after throwing an instance of 'libmaus2::exception::LibMausException'
what(): failure in ::libmaus2::util::NumberSerialisation::serialiseNumber()
/ibex/scratch/projects/c2014/bcbio2/anaconda/share/biobambam-2.0.87-1/bin/../lib/libmaus2.so.2(libmaus2::util::StackTrace::StackTrace()+0x4c)[0x2b8e0a818f8c]
bamsormadup(libmaus2::exception::LibMausException::LibMausException()+0x20)[0x41fc10]
bamsormadup(unsigned long libmaus2::util::NumberSerialisation::serialiseNumbercat /encrypted/e3007/Yang/pon/work/bcbiotx/tmp99vx5o0k/bcbiotx/tmprxatp9zu/LS1P0002-V02-sort.list
36618 Aborted (core dumped) | bamsormadup threads=16 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmp99vx5o0k/bcbiotx/tmprxatp9zu/LS1P0002-V02-sort.bam-bamsormaduptmp indexfilename=/encrypted/e3007/Yang/pon/work/bcbiotx/tmp99vx5o0k/bcbiotx/tmprxatp9zu/LS1P0002-V02-sort.bam.bai > /encrypted/e3007/Yang/pon/work/bcbiotx/tmp99vx5o0k/bcbiotx/tmprxatp9zu/LS1P0002-V02-sort.bam
' returned non-zero exit status 134.
[2021-01-22T13:21Z] cn513-06-r: Unexpected error
Traceback (most recent call last):
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/distributed/ipythontasks.py", line 54, in _setup_logging
yield config
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/distributed/ipythontasks.py", line 324, in delayed_bam_merge
return ipython.zip_args(apply(sample.delayed_bam_merge, args))
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/distributed/ipythontasks.py", line 82, in apply
return object(args, **kwargs)
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/pipeline/sample.py", line 322, in delayed_bam_merge
out_file=cur_out_file)
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/pipeline/merge.py", line 72, in merge_bam_files
None)
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
_do_run(cmd, checks, log_stdout, env=env)
File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; bamcat level=0 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmp6vugg79l/bcbiotx/tmpn9bj1_8b/LS1P0023-V02-sort.bam-bammerge cat /encrypted/e3007/Yang/pon/work/bcbiotx/tmp6vugg79l/bcbiotx/tmpn9bj1_8b/LS1P0023-V02-sort.list | bamsormadup threads=16 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmp6vugg79l/bcbiotx/tmpn9bj1_8b/LS1P0023-V02-sort.bam-bamsormaduptmp indexfilename=/encrypted/e3007/Yang/pon/work/bcbiotx/tmp6vugg79l/bcbiotx/tmpn9bj1_8b/LS1P0023-V02-sort.bam.bai > /encrypted/e3007/Yang/pon/work/bcbiotx/tmp6vugg79l/bcbiotx/tmpn9bj1_8b/LS1P0023-V02-sort.bam
[V] 178628280 09:30:99297000 MemUsage(size=4135.43,rss=2082.25,peak=4199.4) AutoArrayMemUsage(memusage=3254.88,peakmemusage=3380.91,maxmem=1.75922e+13)
[V] 179306496
[V] 179342995 09:33:48030100 MemUsage(size=4135.43,rss=2082.25,peak=4199.4) AutoArrayMemUsage(memusage=3254.88,peakmemusage=3380.91,maxmem=1.75922e+13)
[V] 180355072
[V] 180770272 09:38:18464500 MemUsage(size=4135.43,rss=2082.25,peak=4199.4) AutoArrayMemUsage(memusage=3254.88,peakmemusage=3380.91,maxmem=1.75922e+13)
"""
naumenko-sa
commented
3 years ago Can you also share your pon.yaml? See also some memory tweaks in: #3230 for production-size pureCN runs.
YaungLiu
commented
3 years ago @naumenko-sa, thanks for your reply, here is my pon.yaml file:
""" details:
naumenko-sa
commented
3 years ago Could you please attach it as a file? It is hard to tell whether we see a GitHub formatting error or config error.
There are samples with and without metadata, also it would help if you described how have you created that config.
naumenko-sa
commented
3 years ago Some samples are with
metadata:
batch: pon_build
svclass: controlor
metadata:
batch: pon_build
svclass: tumorand some are without metadata, for example:
- algorithm:
aligner: bwa
svcaller: gatk-cnv
analysis: variant2
description: M-19-5899_LS1P0002-V01-LIV1_i704-i501
files:
- /encrypted/genomics/YangLiu/hepatocellular/merged/M-19-5899_LS1P0002-V01-LIV1_i704-i501_R1.fastq.gz
- /encrypted/genomics/YangLiu/hepatocellular/merged/M-19-5899_LS1P0002-V01-LIV1_i704-i501_R2.fastq.gz
genome_build: hg38
# here new sample starts
- algorithm:
aligner: bwa
svcaller: gatk-cnv
analysis: variant2
description: LS1P0002-V02Could you make sure that all the samples have their metadata defined, or remove the samples without metadata, and re-run then?
YaungLiu
commented
3 years ago @naumenko-sa , thank you, I checked the data and noticed that there are actually some files without meta information, after I removed those files, it works!
Traceback (most recent call last): File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/ibex/scratch/projects/c2014/bcbio2/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) subprocess.CalledProcessError: Command 'set -o pipefail; bamcat level=0 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam-bammerge `cat /encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.list` | bamsormadup threads=16 tmpfile=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam-bamsormaduptmp indexfilename=/encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsvdfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam.bai > /encrypted/e3007/Yang/pon/work/bcbiotx/tmpjsv__dfl/bcbiotx/tmpil0cv6jd/LS1P0010-V01-sort.bam