GATK Recalibration issue

snashraf commented 7 years ago

Hi team

I was trying to run some jobs and seems my jobs stuck at recalibartion step. Specially after this error. [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/home/nsyed/bcbio/anaconda/opt/gatk-3.8/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console... [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/home/nsyed/bcbio/anaconda/opt/gatk-3.8/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

[2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,318 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,317 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-0-ge9d806836, Compiled 2017/07/28 21:26:50 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-0-ge9d806836, Compiled 2017/07/28 21:26:50 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,321 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,321 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,321 HelpFormatter - [Sun Oct 15 15:03:55 AST 2017] Executing on Linux 3.10.0-229.el7.x86_64 amd64 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - [Sun Oct 15 15:03:55 AST 2017] Executing on Linux 3.10.0-229.el7.x86_64 amd64 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,321 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_121-b15 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,320 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_121-b15 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,324 HelpFormatter - Program Args: -I /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bamclean/0200135361/0200135361-noextras-dedup.bam -T BaseRecalibrator -o /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bcbiotx/tmpdVpRwd/0200135361-noextras-dedup-recal.grp -R /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa --knownSites /gpfs/home/nsyed/bcbio/genomes/Hsapiens/GRCh37/variation/dbsnp-150.vcf.gz -L /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bedprep/S07604514_Covered_withoutChr.bed --interval_set_rule INTERSECTION -nct 8 -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,325 HelpFormatter - Program Args: -I /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bamclean/0200135371/0200135371-noextras-dedup.bam -T BaseRecalibrator -o /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bcbiotx/tmpiV2pdB/0200135371-noextras-dedup-recal.grp -R /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa --knownSites /gpfs/home/nsyed/bcbio/genomes/Hsapiens/GRCh37/variation/dbsnp-150.vcf.gz -L /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bedprep/S07604514_Covered_withoutChr.bed --interval_set_rule INTERSECTION -nct 8 -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,327 HelpFormatter - Executing as nsyed@hpcfatnode01.research.sidra.local on Linux 3.10.0-229.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_121-b15. [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,327 HelpFormatter - Date/Time: 2017/10/15 15:03:55 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,327 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,328 HelpFormatter - Executing as nsyed@hpcfatnode01.research.sidra.local on Linux 3.10.0-229.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_121-b15. [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,327 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,328 HelpFormatter - Date/Time: 2017/10/15 15:03:55 [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,328 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: INFO 15:03:55,328 HelpFormatter - ---------------------------------------------------------------------------------- [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/home/nsyed/bcbio/anaconda/opt/gatk-3.8/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console... [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Unable to create class org.apache.logging.log4j.core.impl.Log4jContextFactory specified in jar:file:/gpfs/home/nsyed/bcbio/anaconda/opt/gatk-3.8/GenomeAnalysisTK.jar!/META-INF/log4j-provider.properties [2017-10-15T12:03Z] hpcfatnode01.research.sidra.local: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

But when I tried to run just GATK command on terminal then I am able to run it properly. unset JAVA_HOME && export PATH=/gpfs/home/nsyed/bcbio/anaconda/bin:$PATH && /gpfs/home/nsyed/bcbio/anaconda/bin/gatk -Xms500m -Xmx37g -XX:+UseSerialGC -Djava.io.tmpdir=/gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bcbiotx/tmpdopyKg -I /gpfs/projects/bioinfo/najeeb/withDavide/templ:Q:ateFinalPaired/work/bamclean/0200135411/0200135411-noextras-dedup.bam -T BaseRecalibrator -o /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bcbiotx/tmpdopyKg/0200135411-noextras-dedup-recal.grp -R /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa --downsample_to_fraction 0.730882441367 --downsampling_type ALL_READS --knownSites /gpfs/home/nsyed/bcbio/genomes/Hsapiens/GRCh37/variation/dbsnp-150.vcf.gz -L /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bedprep/S07604514_Covered_withoutChr.bed --interval_set_rule INTERSECTION -nct 8 -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment

Any idea why this happening ? I am using GATK3.8 with latest version ?

chapmanb commented 7 years ago

Thanks for the detailed report and sorry about the issue. Would you be able to post more of the failed traceback so we can identify the causitive issue? Unfortunately the ERROR StatusLogger Log4j messages are not diagnostic; those are always present in working and non working runs and not related to a problem (https://gatkforums.broadinstitute.org/gatk/discussion/10070/gatk-3-8-log4j-error). Is there some of other failure/traceback downstream of this which we could use to identify why they might be stuck? Thanks much for the help debugging.

biocyberman commented 7 years ago

I "fixed" the error with log4j by doing this:

Download the nightly build of GATK 3.8. Extract the tar.bz2 file.
Copy the jar file to the bcbio_nextgen: cp /tmp/gatk/GenomeAnalysisTK.jar /bcbio/anaconda/opt/gatk-3.8/GenomeAnalysisTK.jar

Step 2 bypasses the version mismatch restriction of register-gatk program. @snashraf You might run into the same problem being discussed here: https://github.com/chapmanb/bcbio-nextgen/pull/2114 For a quick fix, you can just edit <bcbio_installed_dir>/anaconda/lib/python2.7/site-packages/bcbio/variation/recalibrate.py like the way I edited the file in the PR.

MBeyens commented 7 years ago

Hi

I think I got more or less the same GATK indel realigner errors, but during the "delayed_bam_merge" process. Could somebody help to fix this? I think the analysis was one week ongoing, I though it was processing, but no output was generated what so ever during this week of run time. See the included files for logging (downsampled).

bcbio-nextgen-debug.log bcbio-nextgen.log bcbio-nextgen-commands.log

Thanks in advance

Matthias

chapmanb commented 7 years ago

Matthias; Sorry about the issue and thank you for the report. I didn't see any error messages in the log files you sent so can't provide specific suggestions but can try to provide some helpful general thoughts:

The log4f messages above are not a diagnostic error. I'm not sure what versions of GATK do and don't produce them but they are spurious and not an issue if you see them.
We generally don't recommend running IndelRealignment (and it's removed in the upcoming GATK4), so setting realign: false for your runs would avoid this extra step that seems to be causing problems here.

The best way to produce a helpful error message is to run bcbio with a single core (bcbio_nextgen.py -n 1) which will hopefully proceed directly to where it left off and give you a useful error message. Happy to provide more specific help with those outputs if turning off realignment does not resolve the issue. Hope this helps.

MBeyens commented 7 years ago

Realignment was the problem. Thank you!

bcbio / bcbio-nextgen

GATK Recalibration issue #2104