search

broadinstitute / gatk

Official code repository for GATK versions 4 and up
https://software.broadinstitute.org/gatk
Other
1.7k stars 591 forks source link

CRAM index inaccessible if CRAM is from cloud but works locally #3669

Closed sooheelee closed 4 years ago

sooheelee commented 7 years ago

For a PrintReads command that requires accessing the index, e.g. when specifying -L intervals, running locally works but running on gs:// CRAM fails even if index is explicitly specified.

Local run (server)

A command that accesses a local CRAM runs fine.

/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
    -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
     -I /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
    -O /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/test_decram_20171002.bam \
    -L chr17
...
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -I /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -O /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/test_decram_20171002.bam -L chr17
14:52:10.763 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 2:52:10 PM EDT] PrintReads  --output /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/test_decram_20171002.bam --intervals chr17 --input /humgen/gsa-hpprojects/dev/shlee/1kg_GRCh38_exome/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 2:52:10 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
14:52:10.875 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
14:52:10.875 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:52:10.875 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:52:10.875 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:52:10.875 INFO  PrintReads - Deflater: IntelDeflater
14:52:10.875 INFO  PrintReads - Inflater: IntelInflater
14:52:10.876 INFO  PrintReads - GCS max retries/reopens: 20
14:52:10.876 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
14:52:10.876 INFO  PrintReads - Initializing engine
14:52:21.901 INFO  IntervalArgumentCollection - Processing 83257441 bp from intervals
14:52:21.917 INFO  PrintReads - Done initializing engine
14:52:22.027 INFO  ProgressMeter - Starting traversal
14:52:22.027 INFO  ProgressMeter -        Current Locus  Elapsed Minutes       Reads Processed     Reads/Minute
14:52:32.033 INFO  ProgressMeter -        chr17:6779805              0.2                494000        2962814.9
14:52:42.035 INFO  ProgressMeter -       chr17:18100301              0.3               1275000        3823661.7
14:52:52.089 INFO  ProgressMeter -       chr17:32183301              0.5               2017000        4025814.2
14:53:02.141 INFO  ProgressMeter -       chr17:38342966              0.7               2500000        3739436.1
14:53:12.267 INFO  ProgressMeter -       chr17:46549838              0.8               3360000        4012818.7
14:53:22.273 INFO  ProgressMeter -       chr17:63099258              1.0               4210000        4192879.1
14:53:30.687 INFO  PrintReads - No reads filtered by: WellformedReadFilter
14:53:30.687 INFO  ProgressMeter -       chr17:83185333              1.1               5250614        4588427.4
14:53:30.687 INFO  ProgressMeter - Traversal complete. Processed 5250614 total reads in 1.1 minutes.
14:53:33.576 INFO  PrintReads - Shutting down engine
[October 5, 2017 2:53:33 PM EDT] org.broadinstitute.hellbender.tools.PrintReads done. Elapsed time: 1.38 minutes.
Runtime.totalMemory()=8385462272

Cloud CRAM

Running just PrintReads without -L intervals succeeds.

/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
    -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
    -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
    -O HG00190_cram.bam
-bash-4.1$ /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
>     -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
>     -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
>     -O HG00190_cram.bam
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -O HG00190_cram.bam
15:05:33.776 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 3:05:33 PM EDT] PrintReads  --output HG00190_cram.bam --input gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 3:05:33 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
15:05:33.887 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
15:05:33.887 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:05:33.887 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:05:33.887 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:05:33.887 INFO  PrintReads - Deflater: IntelDeflater
15:05:33.887 INFO  PrintReads - Inflater: IntelInflater
15:05:33.887 INFO  PrintReads - GCS max retries/reopens: 20
15:05:33.887 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
15:05:33.887 INFO  PrintReads - Initializing engine
15:05:39.011 INFO  PrintReads - Done initializing engine
15:05:39.298 INFO  ProgressMeter - Starting traversal
15:05:39.299 INFO  ProgressMeter -        Current Locus  Elapsed Minutes       Reads Processed     Reads/Minute
15:05:49.302 INFO  ProgressMeter -        chr1:12666181              0.2                578000        3467306.5
15:05:59.302 INFO  ProgressMeter -        chr1:21255922              0.3               1287000        3860613.9
15:06:09.320 INFO  ProgressMeter -        chr1:36027022              0.5               2121000        4239032.7
15:06:19.323 INFO  ProgressMeter -        chr1:52397728              0.7               3017000        4522786.3
15:06:29.323 INFO  ProgressMeter -        chr1:86811190              0.8               4064000        4874460.3
15:06:39.479 INFO  ProgressMeter -       chr1:111761145              1.0               5079000        5063808.6
...

Adding in -L causes the command to error despite the presence of the index within the same folder.

-bash-4.1$ /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
>     -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
>     -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
>     -O HG00190_cram.bam \
>     -L chr17
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -O HG00190_cram.bam -L chr17
14:59:15.760 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 2:59:15 PM EDT] PrintReads  --output HG00190_cram.bam --intervals chr17 --input gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 2:59:15 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
14:59:15.872 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
14:59:15.873 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:59:15.873 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:59:15.873 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:59:15.873 INFO  PrintReads - Deflater: IntelDeflater
14:59:15.873 INFO  PrintReads - Inflater: IntelInflater
14:59:15.873 INFO  PrintReads - GCS max retries/reopens: 20
14:59:15.873 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
14:59:15.873 INFO  PrintReads - Initializing engine
14:59:21.404 INFO  IntervalArgumentCollection - Processing 83257441 bp from intervals
14:59:21.421 INFO  PrintReads - Shutting down engine
[October 5, 2017 2:59:22 PM EDT] org.broadinstitute.hellbender.tools.PrintReads done. Elapsed time: 0.11 minutes.
Runtime.totalMemory()=2129133568
***********************************************************************

A USER ERROR has occurred: Traversal by intervals was requested but some input files are not indexed.
Please index all input files:

samtools index /1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram

***********************************************************************
Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--javaOptions '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.

Still fails with -readIndex specified (.cram.crai OR .crai):

-bash-4.1$ /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
>     -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
>     -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
>     -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai \
>     -O HG00190_cram.bam \
> 
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai -O HG00190_cram.bam
14:59:57.284 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 2:59:57 PM EDT] PrintReads  --output HG00190_cram.bam --input gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 2:59:57 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
14:59:57.393 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
14:59:57.393 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:59:57.393 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:59:57.393 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:59:57.394 INFO  PrintReads - Deflater: IntelDeflater
14:59:57.394 INFO  PrintReads - Inflater: IntelInflater
14:59:57.394 INFO  PrintReads - GCS max retries/reopens: 20
14:59:57.394 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
14:59:57.394 INFO  PrintReads - Initializing engine
^C-bash-4.1$ /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
>     -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
>     -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
>     -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai \
>     -O HG00190_cram.bam \
>     -L chr17
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai -O HG00190_cram.bam -L chr17
15:00:08.140 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 3:00:08 PM EDT] PrintReads  --output HG00190_cram.bam --intervals chr17 --input gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 3:00:08 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
15:00:08.250 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
15:00:08.250 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:00:08.250 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:00:08.250 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:00:08.250 INFO  PrintReads - Deflater: IntelDeflater
15:00:08.250 INFO  PrintReads - Inflater: IntelInflater
15:00:08.250 INFO  PrintReads - GCS max retries/reopens: 20
15:00:08.250 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
15:00:08.250 INFO  PrintReads - Initializing engine
15:00:13.258 INFO  IntervalArgumentCollection - Processing 83257441 bp from intervals
15:00:13.275 INFO  PrintReads - Shutting down engine
[October 5, 2017 3:00:14 PM EDT] org.broadinstitute.hellbender.tools.PrintReads done. Elapsed time: 0.10 minutes.
Runtime.totalMemory()=2233466880
***********************************************************************

A USER ERROR has occurred: Traversal by intervals was requested but some input files are not indexed.
Please index all input files:

samtools index /1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram

***********************************************************************
Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--javaOptions '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.
-bash-4.1$ /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-launch PrintReads \
>     -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram \
>     -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa \
>     -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.crai \
>     -O HG00190_cram.bam \
>     -L chr17
Using GATK jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -jar /humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar PrintReads -I gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram -R /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa -readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.crai -O HG00190_cram.bam -L chr17
15:00:28.170 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/humgen/gsa-hpprojects/GATK/gatk4/gatk-4.beta.5/gatk-package-4.beta.5-local.jar!/com/intel/gkl/native/libgkl_compression.so
[October 5, 2017 3:00:28 PM EDT] PrintReads  --output HG00190_cram.bam --intervals chr17 --input gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram --readIndex gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.crai --reference /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysis_set_plus_decoy_hla.fa  --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --interval_merging_rule ALL --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --gcs_max_retries 20 --disableToolDefaultReadFilters false
[October 5, 2017 3:00:28 PM EDT] Executing as shlee@gsa5.broadinstitute.org on Linux 2.6.32-642.15.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Version: 4.beta.5
15:00:28.284 INFO  PrintReads - HTSJDK Defaults.COMPRESSION_LEVEL : 1
15:00:28.284 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:00:28.284 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:00:28.284 INFO  PrintReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:00:28.284 INFO  PrintReads - Deflater: IntelDeflater
15:00:28.284 INFO  PrintReads - Inflater: IntelInflater
15:00:28.285 INFO  PrintReads - GCS max retries/reopens: 20
15:00:28.285 INFO  PrintReads - Using google-cloud-java patch c035098b5e62cb4fe9155eff07ce88449a361f5d from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
15:00:28.285 INFO  PrintReads - Initializing engine
15:00:33.117 INFO  IntervalArgumentCollection - Processing 83257441 bp from intervals
15:00:33.134 INFO  PrintReads - Shutting down engine
[October 5, 2017 3:00:34 PM EDT] org.broadinstitute.hellbender.tools.PrintReads done. Elapsed time: 0.10 minutes.
Runtime.totalMemory()=2255486976
***********************************************************************

A USER ERROR has occurred: Traversal by intervals was requested but some input files are not indexed.
Please index all input files:

samtools index /1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram

***********************************************************************
Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--javaOptions '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.
-bash-4.1$

Confirm all files present in bucket

WMCF9-CB5:newCNV shlee$ gsutil ls gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN*
gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.bam.bas
gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.crai
gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram
gs://shlee-dev/1kg/exome_GRCh38DH/cram/HG00190.alt_bwamem_GRCh38DH.20150826.FIN.exome.cram.crai
droazen commented 6 years ago

This came up again recently. It looks like HTSJDK discovers the crai index correctly when it's in a bucket, but throws that information away in SamReaderFactory.open() when it calls indexMaybe.asFile(), which returns null

droazen commented 6 years ago

Might actually be a simple fix in HTSJDK...

sooheelee commented 6 years ago

That would be great...

cmnbroad commented 6 years ago

Fix is a few lines of code in the increasingly abominable SamReaderFactory. PR is https://github.com/samtools/htsjdk/pull/1125.

cmnbroad commented 4 years ago

This has been fixed for quite a while. Its the test PR that was controversial.