search

broadinstitute / gatk

Official code repository for GATK versions 4 and up
https://software.broadinstitute.org/gatk
Other
1.68k stars 589 forks source link

The error : Invalid GZIP header when I run the BaseRecalibrator #5968

Closed wuuly closed 4 years ago

wuuly commented 5 years ago

I run the BaseRecalibrator,and at fisrt it can good running,after a time,I got this error： htsjdk.samtools.SAMFormatException: Invalid GZIP header This is the log: Using GATK jar /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar Running: java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx20G -Djava.io.tmpdir=./ -jar /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar BaseRecalibrator -R /data/home/wuly/source/Homo_sapiens_assembly38.fasta -I M1.bam --known-sites /data/home/wuly/source/dbsnp_146.hg38.vcf.gz --known-sites /data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz --known-sites /data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz --known-sites /data/home/wuly/source/hapmap_3.3.hg38.vcf.gz -O M1_recal.table17:55:54.326 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk-package-4.1.2.0-local.jar!/com/intel/gkl/native/libgkl_compre ssion.soMay 24, 2019 5:55:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine INFO: Failed to detect whether we are running on Google Compute Engine. 17:55:56.095 INFO BaseRecalibrator - ------------------------------------------------------------ 17:55:56.096 INFO BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.1.2.0 17:55:56.096 INFO BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/ 17:55:56.096 INFO BaseRecalibrator - Executing as wuly@localhost.localdomain on Linux v3.10.0-957.10.1.el7.x86_64 amd64 17:55:56.096 INFO BaseRecalibrator - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01 17:55:56.096 INFO BaseRecalibrator - Start Date/Time: May 24, 2019 5:55:54 PM EDT 17:55:56.096 INFO BaseRecalibrator - ------------------------------------------------------------ 17:55:56.096 INFO BaseRecalibrator - ------------------------------------------------------------ 17:55:56.096 INFO BaseRecalibrator - HTSJDK Version: 2.19.0 17:55:56.096 INFO BaseRecalibrator - Picard Version: 2.19.0 17:55:56.096 INFO BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2 17:55:56.096 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 17:55:56.097 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 17:55:56.097 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 17:55:56.097 INFO BaseRecalibrator - Deflater: IntelDeflater 17:55:56.097 INFO BaseRecalibrator - Inflater: IntelInflater 17:55:56.097 INFO BaseRecalibrator - GCS max retries/reopens: 20 17:55:56.097 INFO BaseRecalibrator - Requester pays: disabled 17:55:56.097 INFO BaseRecalibrator - Initializing engine 17:55:56.911 INFO FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/dbsnp_146.hg38.vcf.gz 17:55:57.182 INFO FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz 17:55:57.332 INFO FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz 17:55:57.421 INFO FeatureManager - Using codec VCFCodec to read file file:///data/home/wuly/source/hapmap_3.3.hg38.vcf.gz 17:55:57.494 WARN IndexUtils - Feature file "/data/home/wuly/source/dbsnp_146.hg38.vcf.gz" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the as sociated index file17:55:57.630 INFO BaseRecalibrator - Done initializing engine 17:55:57.653 INFO BaseRecalibrationEngine - The covariates being used here: 17:55:57.654 INFO BaseRecalibrationEngine - ReadGroupCovariate 17:55:57.654 INFO BaseRecalibrationEngine - QualityScoreCovariate 17:55:57.654 INFO BaseRecalibrationEngine - ContextCovariate 17:55:57.654 INFO BaseRecalibrationEngine - CycleCovariate 17:55:57.715 INFO ProgressMeter - Starting traversal 17:55:57.716 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute 17:56:07.767 INFO ProgressMeter - chr1:3430732 0.2 311000 1857086.0 17:56:17.858 INFO ProgressMeter - chr1:7301596 0.3 532000 1584827.0 17:56:27.863 INFO ProgressMeter - chr1:11090586 0.5 798000 1588217.7 17:56:37.879 INFO ProgressMeter - chr1:14664558 0.7 1111000 1659777.9 17:56:47.881 INFO ProgressMeter - chr1:17634495 0.8 1461000 1747468.3 17:56:57.887 INFO ProgressMeter - chr1:21468717 1.0 1733000 1728075.0 17:57:07.908 INFO ProgressMeter - chr1:25309233 1.2 2054000 1755780.7 17:57:17.918 INFO ProgressMeter - chr1:28792669 1.3 2400000 1795466.4 17:57:27.966 INFO ProgressMeter - chr1:33012169 1.5 2690000 1788365.7 17:57:37.973 INFO ProgressMeter - chr1:37557597 1.7 2985000 1786426.7 17:57:47.991 INFO ProgressMeter - chr1:42057219 1.8 3294000 1792246.7 17:57:58.015 INFO ProgressMeter - chr1:45421695 2.0 3676000 1833431.7 17:58:08.018 INFO ProgressMeter - chr1:51165712 2.2 3940000 1814246.9 17:58:18.086 INFO ProgressMeter - chr1:55784091 2.3 4248000 1815785.5 17:58:28.116 INFO ProgressMeter - chr1:62578827 2.5 4422000 1764095.7 17:58:38.238 INFO ProgressMeter - chr1:69006155 2.7 4630000 1730603.9 17:58:48.275 INFO ProgressMeter - chr1:76342290 2.8 4796000 1687158.1 17:58:58.291 INFO ProgressMeter - chr1:83911189 3.0 4937000 1640426.4 17:59:08.301 INFO ProgressMeter - chr1:89935260 3.2 5165000 1626046.1 17:59:18.383 INFO ProgressMeter - chr1:96322538 3.3 5382000 1609233.2 17:59:28.438 INFO ProgressMeter - chr1:103371293 3.5 5558000 1582559.0 17:59:38.454 INFO ProgressMeter - chr1:109630766 3.7 5794000 1574898.7 17:59:48.474 INFO ProgressMeter - chr1:114521987 3.8 6102000 1586604.1 17:59:58.487 INFO ProgressMeter - chr1:119925334 4.0 6364000 1585905.3 18:00:08.498 INFO ProgressMeter - chr1:147182812 4.2 6805000 1628107.3 18:00:18.507 INFO ProgressMeter - chr1:151055663 4.3 7257000 1669612.8 18:00:28.517 INFO ProgressMeter - chr1:152309095 4.5 7772000 1722008.9 18:00:38.543 INFO ProgressMeter - chr1:154465127 4.7 8235000 1759446.2 18:00:48.545 INFO ProgressMeter - chr1:156369370 4.8 8736000 1802296.2 18:00:58.559 INFO ProgressMeter - chr1:158847678 5.0 9182000 1831254.2 18:01:08.564 INFO ProgressMeter - chr1:161306195 5.2 9619000 1856663.1 18:01:18.596 INFO ProgressMeter - chr1:166221065 5.3 9885000 1848360.3 18:01:28.608 INFO ProgressMeter - chr1:170690192 5.5 10191000 1847914.1 18:01:38.620 INFO ProgressMeter - chr1:175385996 5.7 10496000 1847323.6 18:01:48.623 INFO ProgressMeter - chr1:180166382 5.8 10795000 1845788.2 18:01:58.639 INFO ProgressMeter - chr1:184568326 6.0 11126000 1849596.3 18:02:08.659 INFO ProgressMeter - chr1:189989846 6.2 11368000 1838773.1 18:02:18.685 INFO ProgressMeter - chr1:196788442 6.3 11527000 1815428.1 18:02:28.690 INFO ProgressMeter - chr1:201317675 6.5 11862000 1820376.8 18:02:38.693 INFO ProgressMeter - chr1:204176575 6.7 12290000 1839012.8 18:02:48.701 INFO ProgressMeter - chr1:207325661 6.8 12708000 1855250.2 18:02:58.737 INFO ProgressMeter - chr1:211941783 7.0 13001000 1852781.7 18:03:08.789 INFO ProgressMeter - chr1:217052843 7.2 13270000 1847019.0 18:03:18.840 INFO ProgressMeter - chr1:222942848 7.4 13509000 1837446.7 18:03:28.843 INFO ProgressMeter - chr1:227016956 7.5 13856000 1842855.4 18:03:38.858 INFO ProgressMeter - chr1:230704130 7.7 14213000 1849294.6 18:03:48.900 INFO ProgressMeter - chr1:235326795 7.9 14501000 1846539.8 18:03:58.915 INFO ProgressMeter - chr1:239911899 8.0 14790000 1844143.5 18:04:08.930 INFO ProgressMeter - chr1:246522306 8.2 15003000 1832561.8 18:04:17.556 INFO BaseRecalibrator - Shutting down engine [May 24, 2019 6:04:17 PM EDT] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 8.39 minutes. Runtime.totalMemory()=4407164928 htsjdk.samtools.SAMFormatException: Invalid GZIP header at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:121) at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96) at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550) at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532) at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468) at htsjdk.samtools.util.BlockCompressedInputStream.seek(BlockCompressedInputStream.java:380) at htsjdk.tribble.readers.TabixReader$IteratorImpl.next(TabixReader.java:427) at htsjdk.tribble.readers.TabixIteratorLineReader.readLine(TabixIteratorLineReader.java:46) at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170) at htsjdk.tribble.TabixFeatureReader$FeatureIterator.(TabixFeatureReader.java:159) at htsjdk.tribble.TabixFeatureReader.query(TabixFeatureReader.java:133) at org.broadinstitute.hellbender.engine.FeatureDataSource.refillQueryCache(FeatureDataSource.java:533) at org.broadinstitute.hellbender.engine.FeatureDataSource.queryAndPrefetch(FeatureDataSource.java:502) at org.broadinstitute.hellbender.engine.FeatureManager.getFeatures(FeatureManager.java:340) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:172) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:124) at org.broadinstitute.hellbender.engine.FeatureContext.getValues(FeatureContext.java:262) at org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator.apply(BaseRecalibrator.java:185) at org.broadinstitute.hellbender.engine.ReadWalker.lambda$traverse$0(ReadWalker.java:96) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.broadinstitute.hellbender.engine.ReadWalker.traverse(ReadWalker.java:94) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1039) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205) at org.broadinstitute.hellbender.Main.main(Main.java:291)

My GATK version is :GATK4.1.2.0 My command is: /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk --java-options "-Xmx20G -Djava.io.tmpdir=./" BaseRecalibrator -R /data/home/wuly/source/Homo_sapiens_assembly38.fasta \ -I M1.bam \ --known-sites /data/home/wuly/source/hapmap_3.3.hg38.vcf.gz \ --known-sites /data/home/wuly/source/dbsnp_146.hg38.vcf.gz \ --known-sites /data/home/wuly/source/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \ --known-sites /data/home/wuly/source/1000G_phase1.snps.high_confidence.hg38.vcf.gz \ -O M1_recal.table Then I run the ValidateSamFile to check the BAM file,this is the command : /data/home/wuly/soft/GATK4/gatk-4.1.2.0/gatk --java-options "-Xmx20G -Djava.io.tmpdir=./" ValidateSamFile -I M1.bam

And the result is: No errors found I also tried to use the BAM file before I merge them to run BaseRecalibrator and ValidateSamFile, but I got the same result.Can anybody tell me how solve this problem?

ldgauthier commented 4 years ago

@wuuly you can report if you are you still interested in having this resolved. If so, being able to reproduce the issue would be very helpful: https://gatk.broadinstitute.org/hc/en-us/articles/360035889671