Error encountered while running on downsampled BAM

[snakesch]

Dear Developers,

Thank you very much for the amazing tool! I recently encountered an issue while running DeepVariant on a GIAB sample. I managed to run DeepVariant open another sample successfully so I believe there may be something wrong in the current sample input. It would be most helpful and appreciated if you could kindly take a look at the error messages.

Setup

• Operating system: CentOS 7 x86_64
• DeepVariant version: 1.6.1
• Installation method (Docker, built from source, etc.): Singularity (v3.10.0)
• Type of data: (sequencing instrument, reference genome, anything special that is unlike the case studies?) Input BAM was downsampled 10-fold to 30X.

Steps to reproduce:

• Command:

  singularity run \
      -B /usr/lib/locale/:/usr/lib/locale/ \
      -B /paedyl01/disk1/louisshe/ref/GIAB/HG005/hs37d5/novoalign_bam/:/input_reads \
      -B /paedyl01/disk1/louisshe/out/GIAB/HG005/heterozygous_deletions/heterozygous_sites/:/output \
      -B /tmp:/tmp \
      -B /paedyl01/disk1/louisshe/ref/hs37d5:/ref/hs37d5 \
      -B /paedyl01/disk1/louisshe/ref/hg19:/ref/hg19 \
      --home /paedyl01/disk1/louisshe/ref/GIAB/HG005/hs37d5/ \
      --contain \
      /paedyl01/disk1/louisshe/tools/DeepVariant/deepvariant_1.6.1.sif \
      /opt/deepvariant/bin/run_deepvariant \
        --model_type=WGS \
        --ref=/ref/hs37d5/hs37d5.fa \
        --reads=/input_reads/HG005.hs37d5.30x.bam \
        --output_vcf=/output/HG005.dv.vcf.gz \
        --output_gvcf=/output/HG005.dv.g.vcf.gz \
        --num_shards=10 \
        --intermediate_results_dir=/tmp \
        --logging_dir=/output/log \
        --dry_run=false \
        --par_regions_bed=/ref/hg19/ucsc.hg19.par.bed \
        --haploid_contigs="chrX,chrY"

• Error trace: Error trace below is from HG005_deppvariant.log. No error prompts prior to this step.

  
  ***** Running the command:*****
  time /opt/deepvariant/bin/call_variants --outfile "/tmp/call_variants_output.tfrecord.gz" --examples "/tmp/make_examp

/usr/local/lib/python3.8/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:

TensorFlow Addons (TFA) has ended development and introduction of new features. TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Ker

For more information see: https://github.com/tensorflow/addons/issues/2807

warnings.warn( I0619 14:57:56.059498 47403021002560 call_variants.py:563] Total 1 writing processes started. I0619 14:57:56.063244 47403021002560 dv_utils.py:370] From /tmp/make_examples.tfrecord-00000-of-00010.gz.example_info I0619 14:57:56.063441 47403021002560 call_variants.py:588] Shape of input examples: [100, 221, 7] I0619 14:57:56.063909 47403021002560 call_variants.py:592] Use saved model: True 2024-06-19 14:57:57.916727: F tensorflow/tsl/platform/env.cc:391] Check failed: -1 != path_length (-1 vs. -1) Fatal Python error: Aborted

Current thread 0x00002b1ce03a6740 (most recent call first): File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/importer.py", line 500 in _import_graph_de File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/importer.py", line 414 in import_graph_def File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/function_def_to_graph.py", line 87 in func File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/function_deserialization.py", line 416 i File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 154 in init File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 958 in load_partial File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 828 in load File "/tmp/Bazel.runfiles_vitt1d55/runfiles/com_google_deepvariant/deepvariant/callvariants.py", line 596 in call File "/tmp/Bazel.runfiles_vitt1d55/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 768 in main File "/tmp/Bazel.runfiles_vitt1d55/runfiles/absl_py/absl/app.py", line 258 in _run_main File "/tmp/Bazel.runfiles_vitt1d55/runfiles/absl_py/absl/app.py", line 312 in run File "/tmp/Bazel.runfiles_vitt1d55/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 789 in <modu

real 0m5.038s user 0m3.921s sys 0m1.122s Process ForkProcess-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/tmp/Bazel.runfiles_vitt1d55/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 454, in post File "/usr/lib/python3.8/multiprocessing/queues.py", line 108, in get raise Empty _queue.Empty INFO: Cleaning up image...

Please let me know if I could provide the input BAM for testing/debugging. Thank you for your time.

Best regards,
Louis

[kishwarshafin]

@snakesch can you please send the file to shafin@google.com. It's unclear from the output what exactly is happening, are you certain make_examples finished properly? Anyways, if you send me the file, I can run on my end to see if there's an issue with the pipeline or the file.

[kishwarshafin]

Hi @snakesch , pinging one more time to see if its possible for you to share the bam.

[snakesch]

Hi Kishwar,

Apologies for the delayed response. I later discovered the issue was due to CPU memory issue, not with DeepVariant. I will close the issue for now, sorry for the oversight.

Best, Louis