search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
981 stars 355 forks source link

Mutect2 error with GATK 4.1.2.0 in specific region #3039

Closed chenzler closed 4 years ago

chenzler commented 4 years ago

Hi,

I've been running bcbio on a sample with matched tumor/normal (using the variant2 analysis pipeline), and I keep getting the following error on one specific region of the genome. Mutect2 runs for that region, and it looks like it's successful, but then it throws an error. If I run the individual problematic command (as listed in the error report), it runs to completion cleanly. Any ideas what's causing the problem?

Here's the output & error, starting with the end of the Mutect2 run for that region:

23:59:12.710 INFO  ProgressMeter -       chr7:130099877            194.4                146190            751.9
23:59:22.804 INFO  ProgressMeter -       chr7:130130134            194.6                146310            751.9
23:59:31.364 INFO  Mutect2 - 10170021 read(s) filtered by: (((((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND M
appedReadFilter) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonChimericOriginalAlignmentReadFilter) AND NonZer
oReferenceLengthAlignmentReadFilter) AND ReadLengthReadFilter) AND GoodCigarReadFilter) AND WellformedReadFilter)
  10170021 read(s) filtered by: ((((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilter) AND NotSeco
ndaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonChimericOriginalAlignmentReadFilter) AND NonZeroReferenceLengthAlignmentRea
dFilter) AND ReadLengthReadFilter) AND GoodCigarReadFilter)
      10170021 read(s) filtered by: (((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilter) AND NotS
econdaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonChimericOriginalAlignmentReadFilter) AND NonZeroReferenceLengthAlignment
ReadFilter) AND ReadLengthReadFilter)
          10170021 read(s) filtered by: ((((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilter) AND N
otSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonChimericOriginalAlignmentReadFilter) AND NonZeroReferenceLengthAlignm
entReadFilter)
              10170021 read(s) filtered by: (((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilter) AN
D NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter) AND NonChimericOriginalAlignmentReadFilter)
                  10170021 read(s) filtered by: ((((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilter)
 AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter) AND PassesVendorQualityCheckReadFilter)
                      10170021 read(s) filtered by: (((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFilt
er) AND NotSecondaryAlignmentReadFilter) AND NotDuplicateReadFilter)
                          1149988 read(s) filtered by: ((((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedReadFi
lter) AND NotSecondaryAlignmentReadFilter)
                              1129700 read(s) filtered by: (((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter) AND MappedRea
dFilter)
                                  1129700 read(s) filtered by: ((MappingQualityReadFilter AND MappingQualityAvailableReadFilter) AND MappingQualityNotZeroReadFilter)
                                      1129700 read(s) filtered by: (MappingQualityReadFilter AND MappingQualityAvailableReadFilter)
                                          1129700 read(s) filtered by: MappingQualityReadFilter 
                              20288 read(s) filtered by: NotSecondaryAlignmentReadFilter 
                          9020033 read(s) filtered by: NotDuplicateReadFilter 
23:59:31.364 INFO  ProgressMeter -       chr7:130152190            194.7                146408            751.8
23:59:31.364 INFO  ProgressMeter - Traversal complete. Processed 146408 total regions in 194.7 minutes.
23:59:31.463 INFO  VectorLoglessPairHMM - Time spent in setup for JNI call : 6.913561473000001
23:59:31.463 INFO  PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 7147.81715048
23:59:31.463 INFO  SmithWatermanAligner - Total compute time in java Smith-Waterman : 1876.24 sec
23:59:31.464 INFO  Mutect2 - Shutting down engine
[December 10, 2019 11:59:31 PM CST] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 194.80 minutes.
Runtime.totalMemory()=691404800
Tool returned:
SUCCESS
pure virtual method called
terminate called without an active exception
Using GATK jar /home/public/bcbio/stable/anaconda/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms681m -Xm
x1727m -XX:+UseSerialGC -Djava.io.tmpdir=/home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig -jar /home/public/bcbio/stable/ana
conda/share/gatk4-4.1.2.0-1/gatk-package-4.1.2.0-local.jar Mutect2 --annotation ClippingRankSumTest --annotation DepthPerSampleHC --reference /home/public/bc
bio/stable/genomes/Hsapiens/hg19/seq/hg19.fa --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --a
nnotation RMSMappingQuality --annotation FisherStrand --annotation MappingQuality --annotation DepthPerAlleleBySample --annotation Coverage --read-validation-stringency LENIENT
 -I /home/msistaff/ch_user/sample1/sample1_mangi/work/align/SAMPLE1_SK_100X_S1/SAMPLE1_SK_100X_S1-sort.bam --tumor-sample SAMPLE1_SK_100X_S1 -I /home/msist
aff/ch_user/sample1/sample1_mangi/work/align/SAMPLE1_SW_20X_S2/SAMPLE1_SW_20X_S2-sort.bam --normal-sample SAMPLE1_SW_20X_S2 -L /home/msistaff/ch_user/sample1/sample1_mangi/w
ork/mutect2/chr7/SAMPLE1-chr7_93418103_130154524-regions.bed --interval-set-rule INTERSECTION -O /home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig
/SAMPLE1-chr7_93418103_130154524-raw.vcf.gz
' returned non-zero exit status 250.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/public/bcbio/bcbio_tooldir/bin/bcbio_nextgen.py", line 238, in <module>
    main(**kwargs)
  File "/home/public/bcbio/bcbio_tooldir/bin/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 45, in run_main
    fc_dir, run_info_yaml)
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 89, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 152, in variant2pipeline
    samples = genotype.parallel_variantcall_region(samples, run_parallel)
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/variation/genotype.py", line 208, in parallel_variantcall_region
    "vrn_file", ["region", "sam_ref", "config"]))
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/distributed/split.py", line 35, in grouped_parallel_split_combine
    final_output = parallel_fn(parallel_name, split_args)
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/public/bcbio/stable/anaconda/lib/python3.6/multiprocessing/pool.py", line 670, in get
    raise self._value
subprocess.CalledProcessError: Command 'set -o pipefail; unset JAVA_HOME && export PATH=/home/public/bcbio/stable/anaconda/bin:"$PATH" && gatk --java-options '-Xms681m -Xmx1727m -XX:+UseSerialGC -Djava.io.tmpdir=/home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig' Mutect2 --annotation ClippingRankSumTest --annotation DepthPerSampleHC --reference /home/public/bcbio/stable/genomes/Hsapiens/hg19/seq/hg19.fa --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --annotation RMSMappingQuality --annotation FisherStrand --annotation MappingQuality --annotation DepthPerAlleleBySample --annotation Coverage --read-validation-stringency LENIENT -I /home/msistaff/ch_user/sample1/sample1_mangi/work/align/SAMPLE1_SK_100X_S1/SAMPLE1_SK_100X_S1-sort.bam --tumor-sample SAMPLE1_SK_100X_S1 -I /home/msistaff/ch_user/sample1/sample1_mangi/work/align/SAMPLE1_SW_20X_S2/SAMPLE1_SW_20X_S2-sort.bam --normal-sample SAMPLE1_SW_20X_S2 -L /home/msistaff/ch_user/sample1/sample1_mangi/work/mutect2/chr7/SAMPLE1-chr7_93418103_130154524-regions.bed --interval-set-rule INTERSECTION -O /home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig/SAMPLE1-chr7_93418103_130154524-raw.vcf.gz && unset JAVA_HOME && export PATH=/home/public/bcbio/stable/anaconda/bin:"$PATH" && gatk --java-options '-Xms681m -Xmx1727m -XX:+UseSerialGC -Djava.io.tmpdir=/home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig' FilterMutectCalls --reference /home/public/bcbio/stable/genomes/Hsapiens/hg19/seq/hg19.fa --variant /home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig/SAMPLE1-chr7_93418103_130154524-raw.vcf.gz --output /home/msistaff/ch_user/sample1/sample1_mangi/work/bcbiotx/tmpioe2vbig/SAMPLE1-chr7_93418103_130154524-raw-filt.vcf.gz
roryk commented 4 years ago

Hi @chenzler,

Hm-- I'm not sure what is happening here, I can't see anything super useful from the error messages, it is weird it works outside of bcbio but fails in the same spot in bcbio. I'm wondering if this is maybe a memory issue, when you are running it outside of bcbio are you still passing it the same amount of memory? If we can figure out something that we can reproduce we can try to dig into what is causing the problem. You can try giving GATK more memory by setting:

  gatk:
    jvm_opts: ["-Xms500m", "-Xmx3500m"]

under the resources section in your bcbio_system.yaml file, which should give GATK more memory to work with.

chenzler commented 4 years ago

Well, this is interesting. I tried giving GATK more memory, and it caught on a different region (on chr1). I had previously tried changing the general default memory settings to give the whole process more memory; I assumed that would also give more memory to GATK, but do I need to allocate specifically to GATK as well? In all my previous runs, I tweaked the general default memory settings in the sample.yaml file, and no matter what I did it failed on the same region of chr7. But changing the allocation for GATK specifically changed that, which seems to suggest that memory is causing the problem at some level. I'll try giving even more memory specifically to GATK and see what happens... Thanks!

naumenko-sa commented 4 years ago

closing for now, let us know if you still have issues!