search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
994 stars 354 forks source link

Error at Deepvariant #3048

Closed kokyriakidis closed 4 years ago

kokyriakidis commented 4 years ago

I cannot get Deepvariant work in any of my data. I have 84x WGS Novaseq 6000 data with very good quality but I get this error.

[2019-12-27T01:12Z] Timing: variant calling
[2019-12-27T01:12Z] multiprocessing: variantcall_sample
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computers / CPU cores / Max jobs to run
[2019-12-27T01:12Z] 1:local / 20 / 1
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2019-12-27T01:12Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.722054 140638928250688 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
[2019-12-27T01:12Z] min_base_quality: 10
[2019-12-27T01:12Z] min_base_quality_mode: ENFORCED_BY_CLIENT
[2019-12-27T01:12Z] I1227 03:12:29.774111 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:29.810076 140638928250688 make_examples.py:1324] Preparing inputs
[2019-12-27T01:12Z] I1227 03:12:29.879332 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:30.626386 140638928250688 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
[2019-12-27T01:12Z] I1227 03:12:30.895529 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-regions.bed with NativeBedReader
[2019-12-27T01:12Z] I1227 03:12:31.032716 140638928250688 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmp05e0_k70/EVANGELIA.tfrecord-00000-of-00001.gz
[2019-12-27T01:12Z] I1227 03:12:31.045903 140638928250688 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[2019-12-27T01:12Z] 2019-12-27 03:12:31.046123: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
[2019-12-27T01:12Z] I1227 03:12:31.122625 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:31.155292 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] Traceback (most recent call last):
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
[2019-12-27T01:12Z]     tf.app.run()
[2019-12-27T01:12Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
[2019-12-27T01:12Z]     _sys.exit(main(argv))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
[2019-12-27T01:12Z]     make_examples_runner(options)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
[2019-12-27T01:12Z]     candidates, examples, gvcfs = region_processor.process(region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
[2019-12-27T01:12Z]     self.in_memory_sam_reader.replace_reads(self.region_reads(region))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
[2019-12-27T01:12Z]     _, reads = self.realigner.realign_reads(reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
[2019-12-27T01:12Z]     self.config.ws_config, self.ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
[2019-12-27T01:12Z]     candidates = _candidates_from_reads(config, ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
[2019-12-27T01:12Z]     expanded_region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
[2019-12-27T01:12Z]     allele_counter, model_conf))
[2019-12-27T01:12Z] TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
[2019-12-27T01:12Z] parallel: This job failed:
[2019-12-27T01:12Z] /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmp05e0_k70/EVANGELIA.tfrecord@1.gz --task 0
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.733736 140036795238208 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
[2019-12-27T01:12Z] min_base_quality: 10
[2019-12-27T01:12Z] min_base_quality_mode: ENFORCED_BY_CLIENT
[2019-12-27T01:12Z] I1227 03:12:29.781769 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:29.810056 140036795238208 make_examples.py:1324] Preparing inputs
[2019-12-27T01:12Z] I1227 03:12:29.879297 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:30.634991 140036795238208 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
[2019-12-27T01:12Z] I1227 03:12:30.903136 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-regions.bed with NativeBedReader
[2019-12-27T01:12Z] I1227 03:12:30.982568 140036795238208 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpci8ke5i3/EVANGELIA.tfrecord-00000-of-00001.gz
[2019-12-27T01:12Z] I1227 03:12:30.994398 140036795238208 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[2019-12-27T01:12Z] 2019-12-27 03:12:30.994606: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
[2019-12-27T01:12Z] I1227 03:12:31.070326 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:31.103122 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] Traceback (most recent call last):
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
[2019-12-27T01:12Z]     tf.app.run()
[2019-12-27T01:12Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
[2019-12-27T01:12Z]     _sys.exit(main(argv))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
[2019-12-27T01:12Z]     make_examples_runner(options)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
[2019-12-27T01:12Z]     candidates, examples, gvcfs = region_processor.process(region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
[2019-12-27T01:12Z]     self.in_memory_sam_reader.replace_reads(self.region_reads(region))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
[2019-12-27T01:12Z]     _, reads = self.realigner.realign_reads(reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
[2019-12-27T01:12Z]     self.config.ws_config, self.ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
[2019-12-27T01:12Z]     candidates = _candidates_from_reads(config, ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
[2019-12-27T01:12Z]     expanded_region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
[2019-12-27T01:12Z]     allele_counter, model_conf))
[2019-12-27T01:12Z] TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
[2019-12-27T01:12Z] parallel: This job failed:
[2019-12-27T01:12Z] /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpci8ke5i3/EVANGELIA.tfrecord@1.gz --task 0
[2019-12-27T01:12Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-regions.bed --logdir /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-work/log --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmp05e0_k70 --sample EVANGELIA
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.722054 140638928250688 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT
I1227 03:12:29.774111 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:29.810076 140638928250688 make_examples.py:1324] Preparing inputs
I1227 03:12:29.879332 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:30.626386 140638928250688 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
I1227 03:12:30.895529 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-regions.bed with NativeBedReader
I1227 03:12:31.032716 140638928250688 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmp05e0_k70/EVANGELIA.tfrecord-00000-of-00001.gz
I1227 03:12:31.045903 140638928250688 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-27 03:12:31.046123: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1227 03:12:31.122625 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.155292 140638928250688 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
    tf.app.run()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
    expanded_region)
  File "/tmp/Bazel.runfiles_WCcEzG/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chr4/EVANGELIA-chr4_136294977_155538272-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmp05e0_k70/EVANGELIA.tfrecord@1.gz --task 0
' returned non-zero exit status 1.
[2019-12-27T01:12Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-regions.bed --logdir /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-work/log --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpci8ke5i3 --sample EVANGELIA
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.733736 140036795238208 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT
I1227 03:12:29.781769 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:29.810056 140036795238208 make_examples.py:1324] Preparing inputs
I1227 03:12:29.879297 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:30.634991 140036795238208 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
I1227 03:12:30.903136 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-regions.bed with NativeBedReader
I1227 03:12:30.982568 140036795238208 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpci8ke5i3/EVANGELIA.tfrecord-00000-of-00001.gz
I1227 03:12:30.994398 140036795238208 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-27 03:12:30.994606: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1227 03:12:31.070326 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.103122 140036795238208 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
    tf.app.run()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
    expanded_region)
  File "/tmp/Bazel.runfiles_mAXZK7/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrX/EVANGELIA-chrX_16378240_36466901-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpci8ke5i3/EVANGELIA.tfrecord@1.gz --task 0
' returned non-zero exit status 1.
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
[2019-12-27T01:12Z] DeepVariant make_examples EVANGELIA
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.707041 139727174838080 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
[2019-12-27T01:12Z] min_base_quality: 10
[2019-12-27T01:12Z] min_base_quality_mode: ENFORCED_BY_CLIENT
[2019-12-27T01:12Z] I1227 03:12:29.756525 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:29.810548 139727174838080 make_examples.py:1324] Preparing inputs
[2019-12-27T01:12Z] I1227 03:12:29.907680 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:31.479063 139727174838080 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
[2019-12-27T01:12Z] I1227 03:12:31.807117 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed with NativeBedReader
[2019-12-27T01:12Z] I1227 03:12:31.864197 139727174838080 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord-00000-of-00001.gz
[2019-12-27T01:12Z] I1227 03:12:31.875258 139727174838080 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
[2019-12-27T01:12Z] 2019-12-27 03:12:31.875533: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
[2019-12-27T01:12Z] I1227 03:12:31.955676 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] I1227 03:12:31.989100 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
[2019-12-27T01:12Z] Traceback (most recent call last):
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
[2019-12-27T01:12Z]     tf.app.run()
[2019-12-27T01:12Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
[2019-12-27T01:12Z]     _sys.exit(main(argv))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
[2019-12-27T01:12Z]     make_examples_runner(options)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
[2019-12-27T01:12Z]     candidates, examples, gvcfs = region_processor.process(region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
[2019-12-27T01:12Z]     self.in_memory_sam_reader.replace_reads(self.region_reads(region))
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
[2019-12-27T01:12Z]     _, reads = self.realigner.realign_reads(reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
[2019-12-27T01:12Z]     self.config.ws_config, self.ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
[2019-12-27T01:12Z]     candidates = _candidates_from_reads(config, ref_reader, reads, region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
[2019-12-27T01:12Z]     expanded_region)
[2019-12-27T01:12Z]   File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
[2019-12-27T01:12Z]     allele_counter, model_conf))
[2019-12-27T01:12Z] TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
[2019-12-27T01:12Z] parallel: This job failed:
[2019-12-27T01:12Z] /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord@1.gz --task 0
[2019-12-27T01:12Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --logdir /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-work/log --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i --sample EVANGELIA
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.707041 139727174838080 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT
I1227 03:12:29.756525 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:29.810548 139727174838080 make_examples.py:1324] Preparing inputs
I1227 03:12:29.907680 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.479063 139727174838080 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
I1227 03:12:31.807117 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed with NativeBedReader
I1227 03:12:31.864197 139727174838080 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord-00000-of-00001.gz
I1227 03:12:31.875258 139727174838080 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-27 03:12:31.875533: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1227 03:12:31.955676 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.989100 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
    tf.app.run()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
    expanded_region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord@1.gz --task 0
' returned non-zero exit status 1.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 600, in __call__
    return self.func(*args, **kwargs)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 256, in __call__
    for func, args, kwargs in self.items]
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 256, in <listcomp>
    for func, args, kwargs in self.items]
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 315, in variantcall_sample
    return genotype.variantcall_sample(*args)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/genotype.py", line 377, in variantcall_sample
    out_file = caller_fn(align_bams, items, ref_file, assoc_files, region, out_file)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/deepvariant.py", line 24, in run
    region, out_file)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/deepvariant.py", line 32, in _run_germline
    example_dir = _make_examples(bam_file, data, ref_file, region_bed, out_file, work_dir)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/deepvariant.py", line 52, in _make_examples
    do.run(cmd, "DeepVariant make_examples %s" % dd.get_sample_name(data))
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --logdir /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-work/log --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i --sample EVANGELIA
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.707041 139727174838080 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT
I1227 03:12:29.756525 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:29.810548 139727174838080 make_examples.py:1324] Preparing inputs
I1227 03:12:29.907680 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.479063 139727174838080 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
I1227 03:12:31.807117 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed with NativeBedReader
I1227 03:12:31.864197 139727174838080 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord-00000-of-00001.gz
I1227 03:12:31.875258 139727174838080 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-27 03:12:31.875533: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1227 03:12:31.955676 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.989100 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
    tf.app.run()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
    expanded_region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord@1.gz --task 0
' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/bin/bcbio_nextgen.py", line 245, in <module>
    main(**kwargs)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/bin/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 154, in variant2pipeline
    samples = genotype.parallel_variantcall_region(samples, run_parallel)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/genotype.py", line 208, in parallel_variantcall_region
    "vrn_file", ["region", "sam_ref", "config"]))
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/split.py", line 35, in grouped_parallel_split_combine
    final_output = parallel_fn(parallel_name, split_args)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 1016, in __call__
    self.retrieve()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 908, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/multiprocessing/pool.py", line 670, in get
    raise self._value
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --logdir /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-work/log --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i --sample EVANGELIA
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s I1227 03:12:29.707041 139727174838080 make_examples.py:377] ReadRequirements are: min_mapping_quality: 10
min_base_quality: 10
min_base_quality_mode: ENFORCED_BY_CLIENT
I1227 03:12:29.756525 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:29.810548 139727174838080 make_examples.py:1324] Preparing inputs
I1227 03:12:29.907680 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.479063 139727174838080 make_examples.py:1248] Common contigs are [u'chr1', u'chr2', u'chr3', u'chr4', u'chr5', u'chr6', u'chr7', u'chr8', u'chr9', u'chr10', u'chr11', u'chr12', u'chr13', u'chr14', u'chr15', u'chr16', u'chr17', u'chr18', u'chr19', u'chr20', u'chr21', u'chr22', u'chrX', u'chrY']
I1227 03:12:31.807117 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed with NativeBedReader
I1227 03:12:31.864197 139727174838080 make_examples.py:1330] Writing examples to /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord-00000-of-00001.gz
I1227 03:12:31.875258 139727174838080 make_examples.py:905] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
2019-12-27 03:12:31.875533: I third_party/nucleus/io/sam_reader.cc:660] Setting HTS_OPT_BLOCK_SIZE to 134217728
I1227 03:12:31.955676 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
I1227 03:12:31.989100 139727174838080 genomics_reader.py:223] Reading /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1461, in <module>
    tf.app.run()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1451, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1341, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1024, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1097, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 610, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 236, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 85, in _candidates_from_reads
    expanded_region)
  File "/tmp/Bazel.runfiles_QuKTAz/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 157, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.9.0-0/binaries/DeepVariant/0.9.0/DeepVariant-0.9.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/REDPRO/work/align/EVANGELIA/EVANGELIA-sort.bam --regions /media/kokyriakidis/REDPRO/work/deepvariant/chrY/EVANGELIA-chrY_16090812_56678023-regions.bed --examples /media/kokyriakidis/REDPRO/work/bcbiotx/tmpb904i_2i/EVANGELIA.tfrecord@1.gz --task 0
' returned non-zero exit status 1.

Any thoughts?

chapmanb commented 4 years ago

Konstantinos; Sorry about the issue, I'm not positive what is happening here. There is a similar issue reported in the DeepVariant issues (https://github.com/google/deepvariant/issues/199) but it seems like in that case it was related to the build process and we're using the pre-compiled DeepVariant from the conda recipe. I'm not positive how to debug further from afar. If you need DeepVariant you could try reverting to a previous version:

bcbio_conda install -n dv -c conda-forge -c bioconda deepvariant=0.8.0

to see if that also has the issue, or try to follow up with the DeepVariant folks to see if they have any tips for debugging this issue. Sorry to not have a better idea but hope this helps.

naumenko-sa commented 4 years ago

closing for now

kokyriakidis commented 4 years ago

@chapmanb @naumenko-sa

Reverting to 0.8 did not solve the error. Is it not working only in me or do you also face this issue?

naumenko-sa commented 4 years ago

I'm not using DeepVariant at the moment.

chapmanb commented 4 years ago

Konstantinos; Sorry about all the problems with this one. I think we finally tracked down the underlying issue, which was due to symlink problems when re-zipping the prepped files after fixing the python references. The latest build 0.10.0_3 should have this fix in and work for any future versions. I hope DeepVariant will be working more cleanly for you now. Thanks again for all the debugging and patience.

kokyriakidis commented 4 years ago

Hi @chapmanb ! Thanks for following on this issue! The update did not fix the problem for me. I get again this error:

[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] Computers / CPU cores / Max jobs to run
[2020-04-03T07:05Z] 1:local / 20 / 1
[2020-04-03T07:05Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2020-04-03T07:05Z] Computers / CPU cores / Max jobs to run
[2020-04-03T07:05Z] 1:local / 20 / 1
[2020-04-03T07:05Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] DeepVariant make_examples THOMAS
[2020-04-03T07:05Z] Computers / CPU cores / Max jobs to run
[2020-04-03T07:05Z] 1:local / 20 / 1
[2020-04-03T07:05Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
[2020-04-03T07:05Z] Computers / CPU cores / Max jobs to run
[2020-04-03T07:05Z] 1:local / 20 / 1
[2020-04-03T07:05Z] Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] lchmod (file attributes) error: Function not implemented
[2020-04-03T07:05Z] Traceback (most recent call last):
[2020-04-03T07:05Z]   File "/tmp/Bazel.runfiles_fdkjijk4/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 44, in <module>
[2020-04-03T07:05Z]     import tensorflow as tf
[2020-04-03T07:05Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 22, in <module>
[2020-04-03T07:05Z]     from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
[2020-04-03T07:05Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
[2020-04-03T07:05Z]     from tensorflow.core.framework.graph_pb2 import *
[2020-04-03T07:05Z]   File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
[2020-04-03T07:05Z]     from google.protobuf import descriptor as _descriptor
[2020-04-03T07:05Z]   File "/tmp/Bazel.runfiles_fdkjijk4/runfiles/com_google_protobuf/python/google/protobuf/descriptor.py", line 47, in <module>
[2020-04-03T07:05Z]     from google.protobuf.pyext import _message
[2020-04-03T07:05Z] ImportError: /RED/BCBIO/RESOURCES/bcbio/anaconda/bin/../lib/libm.so.6: version `GLIBC_2.23' not found (required by /tmp/Bazel.runfiles_fdkjijk4/runfiles/com_google_protobuf/python/google/protobuf/pyext/_message.so)
[2020-04-03T07:05Z] parallel: This job failed:
[2020-04-03T07:05Z] /RED/BCBIO/RESOURCES/bcbio/anaconda/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.10.0-3/binaries/DeepVariant/0.10.0/DeepVariant-0.10.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/prealign/THOMAS/THOMAS-prealign.bam --regions /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chr6/THOMAS-chr6_128999541_149762654-regions-nolcr.bed --examples /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/bcbiotx/tmpjpe84gzw/THOMAS.tfrecord@1.gz --task 0
[2020-04-03T07:05Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/prealign/THOMAS/THOMAS-prealign.bam --regions /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chr6/THOMAS-chr6_128999541_149762654-regions-nolcr.bed --logdir /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chr6/THOMAS-chr6_128999541_149762654-work/log --examples /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/bcbiotx/tmpjpe84gzw --sample THOMAS
----
----
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_g42bk38b/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 44, in <module>
    import tensorflow as tf
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/tmp/Bazel.runfiles_g42bk38b/runfiles/com_google_protobuf/python/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: /RED/BCBIO/RESOURCES/bcbio/anaconda/bin/../lib/libm.so.6: version `GLIBC_2.23' not found (required by /tmp/Bazel.runfiles_g42bk38b/runfiles/com_google_protobuf/python/google/protobuf/pyext/_message.so)
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.10.0-3/binaries/DeepVariant/0.10.0/DeepVariant-0.10.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/prealign/THOMAS/THOMAS-prealign.bam --regions /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chrY/THOMAS-chrY_16091111_56678506-regions-nolcr.bed --examples /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/bcbiotx/tmpwsq_rduk/THOMAS.tfrecord@1.gz --task 0
' returned non-zero exit status 1.

This is a clean run in a new work folder after bcbio upgrade to development and downloading the new conda deepvariant _3 version. I am using UBUNTU 19.10. Have never run deepvariant in an os older than 18.04.

Is this a tensorflow problem? https://github.com/tensorflow/tensorflow/issues/28660

chapmanb commented 4 years ago

Konstantinos; Sorry about the continued problems. Unfortunately we don't have a fix for the glibc issue. This is a separate problem from the initial one, and a limitation of the pre-built binaries, They need to have glibc 2.23 or better on the machine. Unfortunately CentOS 7 and some other common distributions found on compute clusters have older glibc versions than this. Is your compute cluster where this is installed using ubuntu 19.10? That's definitely unexpected to be failing since Ubuntu 18.04 works fine for me with it. I'm not sure how to best debug remotely on this but suggest looking at the glibc versions on the machine you're running this as a first place to sanity check. Hope this helps some.

kokyriakidis commented 4 years ago

Hi @chapmanb 

kokyriakidis@Konstantinos:/media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work$ ldd --version
ldd (Ubuntu GLIBC 2.30-0ubuntu2.1) 2.30
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
kokyriakidis@Konstantinos:/media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work$ /lib/x86_64-linux-gnu/libc.so.6 --version
GNU C Library (Ubuntu GLIBC 2.30-0ubuntu2.1) stable release version 2.30.
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 9.2.1 20191008.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

I am running Ubuntu 19.10.

ImportError: /RED/BCBIO/RESOURCES/bcbio/anaconda/bin/../lib/libm.so.6: version `GLIBC_2.23' not found (required by /tmp/Bazel.runfiles_m10_2gxf/runfiles/com_google_protobuf/python/google/protobuf/pyext/_message.so)

There is not a file like this in this path:

 /RED/BCBIO/RESOURCES/bcbio/anaconda/bin/../lib/libm.so.6
kokyriakidis commented 4 years ago

@chapmanb After a new clean install of bcbio I get another error running deepvariant:

subprocess.CalledProcessError: Command 'dv_make_examples.py --cores 1 --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/prealign/THOMAS/THOMAS-prealign.bam --regions /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chrY/THOMAS-chrY_16091111_56678506-regions-nolcr.bed --logdir /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chrY/THOMAS-chrY_16091111_56678506-work/log --examples /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/bcbiotx/tmppcf9ce06 --sample THOMAS
Computers / CPU cores / Max jobs to run
1:local / 20 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_v9dh1ny9/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 44, in <module>
    import tensorflow as tf
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 101, in <module>
    from tensorflow_core import *
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow_core/__init__.py", line 40, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 947, in _find_and_load_unlocked
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow_core/python/__init__.py", line 68, in <module>
    from tensorflow.core.protobuf.meta_graph_pb2 import TensorInfo
  File "/RED/BCBIO/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/tensorflow_core/core/protobuf/meta_graph_pb2.py", line 526, in <module>
    serialized_options=None, file=DESCRIPTOR),
  File "/tmp/Bazel.runfiles_v9dh1ny9/runfiles/com_google_protobuf/python/google/protobuf/descriptor.py", line 534, in __new__
    return _message.default_pool.FindFieldByName(full_name)
KeyError: "Couldn't find field tensorflow.TensorInfo.CompositeTensor.type_spec"
parallel: This job failed:
/RED/BCBIO/RESOURCES/bcbio/anaconda/bin/python /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv/share/deepvariant-0.10.0-3/binaries/DeepVariant/0.10.0/DeepVariant-0.10.0/make_examples.zip --mode calling --ref /RED/BCBIO/RESOURCES/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --reads /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/prealign/THOMAS/THOMAS-prealign.bam --regions /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/deepvariant/chrY/THOMAS-chrY_16091111_56678506-regions-nolcr.bed --examples /media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work/bcbiotx/tmppcf9ce06/THOMAS.tfrecord@1.gz --task 0
' returned non-zero exit status 1.
chapmanb commented 4 years ago

Konstantinos; Apologies, I'm totally at a loss to explain why you're having so many issues with the DeepVariant binaries and why a re-install changes the glibc issue. You should be getting identical pre-compiled binaries in both cases. I also haven't seeing that tensorflow issue, but am guessing it's a version issue. The main worry I have in the traceback is that it should be using deepvariant from within a separate environment, dv:

bcbio/anaconda/envs/dv/lib/python3.6/site-packages/tensorflow_core

rather than bcbio/anacoda/lib directly. It's really hard to know why it's calling into their remotely but hopefully that gives some clue to help debug the install on your side. Hope this helps some.

kokyriakidis commented 4 years ago

Hi @chapmanb ! Thank you for your help!

Made a new clean install again. The install log for dv env is:

Creating conda environment: dv

## Package Plan ##

  environment location: /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv

  added / updated specs:
    - nomkl
    - python=3.6

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_llvm
  blas               conda-forge/linux-64::blas-2.16-openblas
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.34-h53a641e_0
  libblas            conda-forge/linux-64::libblas-3.8.0-16_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-16_openblas
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-7.3.0-hdf63c60_5
  liblapack          conda-forge/linux-64::liblapack-3.8.0-16_openblas
  liblapacke         conda-forge/linux-64::liblapacke-3.8.0-16_openblas
  libopenblas        conda-forge/linux-64::libopenblas-0.3.9-h5ec1e0e_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2
  llvm-openmp        conda-forge/linux-64::llvm-openmp-9.0.1-hc9558a2_2
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  nomkl              pkgs/main/linux-64::nomkl-3.0-0
  openssl            conda-forge/linux-64::openssl-1.1.1f-h516909a_0
  python             conda-forge/linux-64::python-3.6.10-h9d8adfe_1009_cpython
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h7b6447c_0
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  xz                 conda-forge/linux-64::xz-5.2.4-h516909a_1002
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

# Installing into conda environment dv: deepvariant

## Package Plan ##

  environment location: /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv

  added / updated specs:
    - deepvariant
    - python=3.6

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    altair-4.1.0               |             py_1         614 KB  conda-forge
    boost-1.72.0               |   py36h9de70de_0         339 KB  conda-forge
    boost-cpp-1.72.0           |       h8e57a91_0        21.8 MB  conda-forge
    contextlib2-0.6.0.post1    |             py_0          12 KB  conda-forge
    crcmod-1.7                 |        py36_1002          28 KB  conda-forge
    deepvariant-0.10.0         |   py36h7333d49_3        40.2 MB  bioconda
    enum34-1.1.10              |   py36h9f0ad1d_1           4 KB  conda-forge
    google-cloud-sdk-281.0.0   |   py36h9f0ad1d_2        61.5 MB  conda-forge
    htslib-1.10.2              |       h78d89cc_0         1.7 MB  bioconda
    httplib2-0.17.1            |     pyh9f0ad1d_0          93 KB  conda-forge
    intervaltree-3.0.2         |             py_0          23 KB  conda-forge
    libdeflate-1.3             |       h516909a_0          60 KB  conda-forge
    numpy-1.16.6               |   py36h30dfecb_0          49 KB
    numpy-base-1.16.6          |   py36h2f8d375_0         4.4 MB
    oauth2client-4.1.3         |             py_0          66 KB  conda-forge
    rsa-4.0                    |             py_0          27 KB  conda-forge
    sortedcontainers-2.1.0     |           py36_0          45 KB
    tensorflow-2.0.0           |mkl_py36hef7ec59_0           3 KB
    tensorflow-base-2.0.0      |mkl_py36h9204916_0       100.9 MB
    tensorflow-estimator-2.0.0 |     pyh2649769_0         272 KB
    werkzeug-0.16.1            |             py_0         255 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       232.4 MB

The following NEW packages will be INSTALLED:

  _tflow_select      pkgs/main/linux-64::_tflow_select-2.3.0-mkl
  absl-py            conda-forge/linux-64::absl-py-0.9.0-py36_0
  altair             conda-forge/noarch::altair-4.1.0-py_1
  astor              pkgs/main/linux-64::astor-0.8.0-py36_0
  attrs              conda-forge/noarch::attrs-19.3.0-py_0
  blinker            conda-forge/noarch::blinker-1.4-py_1
  boost              conda-forge/linux-64::boost-1.72.0-py36h9de70de_0
  boost-cpp          conda-forge/linux-64::boost-cpp-1.72.0-h8e57a91_0
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_2
  c-ares             conda-forge/linux-64::c-ares-1.15.0-h516909a_1001
  cachetools         conda-forge/noarch::cachetools-3.1.1-py_0
  certifi            pkgs/main/linux-64::certifi-2019.11.28-py36_1
  cffi               conda-forge/linux-64::cffi-1.14.0-py36hd463f26_0
  chardet            conda-forge/linux-64::chardet-3.0.4-py36h9f0ad1d_1006
  click              conda-forge/noarch::click-7.1.1-pyh8c360ce_0
  contextlib2        conda-forge/noarch::contextlib2-0.6.0.post1-py_0
  crcmod             conda-forge/linux-64::crcmod-1.7-py36_1002
  cryptography       conda-forge/linux-64::cryptography-2.8-py36h45558ae_2
  curl               pkgs/main/linux-64::curl-7.69.1-hbc83047_0
  deepvariant        bioconda/linux-64::deepvariant-0.10.0-py36h7333d49_3
  entrypoints        conda-forge/linux-64::entrypoints-0.3-py36h9f0ad1d_1001
  enum34             conda-forge/linux-64::enum34-1.1.10-py36h9f0ad1d_1
  gast               pkgs/main/linux-64::gast-0.2.2-py36_0
  google-auth        conda-forge/noarch::google-auth-1.12.0-pyh9f0ad1d_0
  google-auth-oauth~ conda-forge/noarch::google-auth-oauthlib-0.4.1-py_2
  google-cloud-sdk   conda-forge/linux-64::google-cloud-sdk-281.0.0-py36h9f0ad1d_2
  google-pasta       conda-forge/noarch::google-pasta-0.2.0-pyh8c360ce_0
  grpcio             pkgs/main/linux-64::grpcio-1.27.2-py36hf8bcb03_0
  h5py               conda-forge/linux-64::h5py-2.10.0-nompi_py36h513d04c_102
  hdf5               conda-forge/linux-64::hdf5-1.10.5-nompi_h3c11f04_1104
  htslib             bioconda/linux-64::htslib-1.10.2-h78d89cc_0
  httplib2           conda-forge/noarch::httplib2-0.17.1-pyh9f0ad1d_0
  icu                conda-forge/linux-64::icu-64.2-he1b5a44_1
  idna               conda-forge/noarch::idna-2.9-py_1
  importlib-metadata conda-forge/linux-64::importlib-metadata-1.6.0-py36h9f0ad1d_0
  importlib_metadata conda-forge/noarch::importlib_metadata-1.6.0-0
  intervaltree       conda-forge/noarch::intervaltree-3.0.2-py_0
  jinja2             conda-forge/noarch::jinja2-2.11.1-py_0
  jsonschema         conda-forge/linux-64::jsonschema-3.2.0-py36h9f0ad1d_1
  keras-applications conda-forge/noarch::keras-applications-1.0.8-py_1
  keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.0-py_1
  krb5               pkgs/main/linux-64::krb5-1.17.1-h173b8e3_0
  libcurl            pkgs/main/linux-64::libcurl-7.69.1-h20c2e04_0
  libdeflate         conda-forge/linux-64::libdeflate-1.3-h516909a_0
  libprotobuf        pkgs/main/linux-64::libprotobuf-3.11.4-hd408876_0
  libssh2            pkgs/main/linux-64::libssh2-1.9.0-h1ba5d50_1
  markdown           conda-forge/noarch::markdown-3.2.1-py_0
  markupsafe         conda-forge/linux-64::markupsafe-1.1.1-py36h8c4c3a4_1
  mock               pkgs/main/noarch::mock-4.0.1-py_0
  numpy              pkgs/main/linux-64::numpy-1.16.6-py36h30dfecb_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.16.6-py36h2f8d375_0
  oauth2client       conda-forge/noarch::oauth2client-4.1.3-py_0
  oauthlib           pkgs/main/noarch::oauthlib-3.1.0-py_0
  openjdk            conda-forge/linux-64::openjdk-8.0.192-h516909a_1004
  opt_einsum         conda-forge/noarch::opt_einsum-3.2.0-py_0
  pandas             conda-forge/linux-64::pandas-1.0.3-py36h830a2c2_0
  parallel           conda-forge/linux-64::parallel-20200322-0
  perl               conda-forge/linux-64::perl-5.26.2-h516909a_1006
  protobuf           conda-forge/linux-64::protobuf-3.11.4-py36h831f99a_1
  psutil             conda-forge/linux-64::psutil-5.7.0-py36h8c4c3a4_1
  pyasn1             conda-forge/noarch::pyasn1-0.4.8-py_0
  pyasn1-modules     conda-forge/noarch::pyasn1-modules-0.2.7-py_0
  pycparser          conda-forge/noarch::pycparser-2.20-py_0
  pyjwt              pkgs/main/linux-64::pyjwt-1.7.1-py36_0
  pyopenssl          conda-forge/noarch::pyopenssl-19.1.0-py_1
  pyrsistent         conda-forge/linux-64::pyrsistent-0.16.0-py36h8c4c3a4_0
  pysocks            conda-forge/linux-64::pysocks-1.7.1-py36h9f0ad1d_1
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  python_abi         conda-forge/linux-64::python_abi-3.6-1_cp36m
  pytz               conda-forge/noarch::pytz-2019.3-py_0
  requests           conda-forge/noarch::requests-2.23.0-pyh8c360ce_2
  requests-oauthlib  pkgs/main/noarch::requests-oauthlib-1.3.0-py_0
  rsa                conda-forge/noarch::rsa-4.0-py_0
  scipy              conda-forge/linux-64::scipy-1.4.1-py36h2d22cac_2
  setuptools         pkgs/main/linux-64::setuptools-46.1.3-py36_0
  six                conda-forge/noarch::six-1.14.0-py_1
  sortedcontainers   pkgs/main/linux-64::sortedcontainers-2.1.0-py36_0
  tensorboard        pkgs/main/noarch::tensorboard-2.1.0-py3_0
  tensorflow         pkgs/main/linux-64::tensorflow-2.0.0-mkl_py36hef7ec59_0
  tensorflow-base    pkgs/main/linux-64::tensorflow-base-2.0.0-mkl_py36h9204916_0
  tensorflow-estima~ pkgs/main/noarch::tensorflow-estimator-2.0.0-pyh2649769_0
  termcolor          conda-forge/noarch::termcolor-1.1.0-py_2
  toolz              conda-forge/noarch::toolz-0.10.0-py_0
  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py36_0
  werkzeug           conda-forge/noarch::werkzeug-0.16.1-py_0
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  wrapt              conda-forge/linux-64::wrapt-1.12.1-py36h8c4c3a4_1
  zipp               conda-forge/noarch::zipp-3.1.0-py_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

I edited my ~/.bashrc file exporting bcbio's bin path:

export PATH=/RED/BCBIO/TOOLS/bin:$PATH

I tried to run my job again but recieved the same error. It uses the libs from the other base env.

Dv env looks fine:

(dv) kokyriakidis@Konstantinos:/media/kokyriakidis/PRODUCTION/BCBIO_RUNS/THOMAS/work$ conda list
# packages in environment at /RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
_tflow_select             2.3.0                       mkl  
absl-py                   0.9.0                    py36_0    conda-forge
altair                    4.1.0                      py_1    conda-forge
astor                     0.8.0                    py36_0  
attrs                     19.3.0                     py_0    conda-forge
blas                      2.16                   openblas    conda-forge
blinker                   1.4                        py_1    conda-forge
boost                     1.72.0           py36h9de70de_0    conda-forge
boost-cpp                 1.72.0               h8e57a91_0    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
c-ares                    1.15.0            h516909a_1001    conda-forge
ca-certificates           2020.1.1                      0  
cachetools                3.1.1                      py_0    conda-forge
certifi                   2019.11.28               py36_1  
cffi                      1.14.0           py36hd463f26_0    conda-forge
chardet                   3.0.4           py36h9f0ad1d_1006    conda-forge
click                     7.1.1              pyh8c360ce_0    conda-forge
contextlib2               0.6.0.post1                py_0    conda-forge
crcmod                    1.7                   py36_1002    conda-forge
cryptography              2.8              py36h45558ae_2    conda-forge
curl                      7.69.1               hbc83047_0  
deepvariant               0.10.0           py36h7333d49_3    bioconda
entrypoints               0.3             py36h9f0ad1d_1001    conda-forge
enum34                    1.1.10           py36h9f0ad1d_1    conda-forge
gast                      0.2.2                    py36_0  
google-auth               1.12.0             pyh9f0ad1d_0    conda-forge
google-auth-oauthlib      0.4.1                      py_2    conda-forge
google-cloud-sdk          281.0.0          py36h9f0ad1d_2    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpcio                    1.27.2           py36hf8bcb03_0  
h5py                      2.10.0          nompi_py36h513d04c_102    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1104    conda-forge
htslib                    1.10.2               h78d89cc_0    bioconda
httplib2                  0.17.1             pyh9f0ad1d_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
idna                      2.9                        py_1    conda-forge
importlib-metadata        1.6.0            py36h9f0ad1d_0    conda-forge
importlib_metadata        1.6.0                         0    conda-forge
intervaltree              3.0.2                      py_0    conda-forge
jinja2                    2.11.1                     py_0    conda-forge
jsonschema                3.2.0            py36h9f0ad1d_1    conda-forge
keras-applications        1.0.8                      py_1    conda-forge
keras-preprocessing       1.1.0                      py_1  
krb5                      1.17.1               h173b8e3_0  
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcurl                   7.69.1               h20c2e04_0  
libdeflate                1.3                  h516909a_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
liblapacke                3.8.0               16_openblas    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libprotobuf               3.11.4               hd408876_0  
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
llvm-openmp               9.0.1                hc9558a2_2    conda-forge
markdown                  3.2.1                      py_0    conda-forge
markupsafe                1.1.1            py36h8c4c3a4_1    conda-forge
mock                      4.0.1                      py_0  
ncurses                   6.1               hf484d3e_1002    conda-forge
nomkl                     3.0                           0  
numpy                     1.16.6           py36h30dfecb_0  
numpy-base                1.16.6           py36h2f8d375_0  
oauth2client              4.1.3                      py_0    conda-forge
oauthlib                  3.1.0                      py_0  
openjdk                   8.0.192           h516909a_1004    conda-forge
openssl                   1.1.1f               h516909a_0    conda-forge
opt_einsum                3.2.0                      py_0    conda-forge
pandas                    1.0.3            py36h830a2c2_0    conda-forge
parallel                  20200322                      0    conda-forge
perl                      5.26.2            h516909a_1006    conda-forge
protobuf                  3.11.4           py36h831f99a_1    conda-forge
psutil                    5.7.0            py36h8c4c3a4_1    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20                       py_0    conda-forge
pyjwt                     1.7.1                    py36_0  
pyopenssl                 19.1.0                     py_1    conda-forge
pyrsistent                0.16.0           py36h8c4c3a4_0    conda-forge
pysocks                   1.7.1            py36h9f0ad1d_1    conda-forge
python                    3.6.10          h9d8adfe_1009_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.6                     1_cp36m    conda-forge
pytz                      2019.3                     py_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.23.0             pyh8c360ce_2    conda-forge
requests-oauthlib         1.3.0                      py_0  
rsa                       4.0                        py_0    conda-forge
scipy                     1.4.1            py36h2d22cac_2    conda-forge
setuptools                46.1.3                   py36_0  
six                       1.14.0                     py_1    conda-forge
sortedcontainers          2.1.0                    py36_0  
sqlite                    3.31.1               h7b6447c_0  
tensorboard               2.1.0                     py3_0  
tensorflow                2.0.0           mkl_py36hef7ec59_0  
tensorflow-base           2.0.0           mkl_py36h9204916_0  
tensorflow-estimator      2.0.0              pyh2649769_0  
termcolor                 1.1.0                      py_2    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
urllib3                   1.25.8                   py36_0  
werkzeug                  0.16.1                     py_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
wrapt                     1.12.1           py36h8c4c3a4_1    conda-forge
xz                        5.2.4             h516909a_1002    conda-forge
zipp                      3.1.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Tensorflow in base bcbio env (/RED/BCBIO/RESOURCES/bcbio/anaconda) is in it's latest version 2.1.0. Are you sure it is a version error?

tensorboard               2.1.0                     py3_0  
tensorflow                2.1.0           mkl_py36h23468d9_0  
tensorflow-base           2.1.0           mkl_py36h6d63fb7_0  
tensorflow-estimator      2.1.0              pyhd54b08b_0

Tensorflow in dv env (/RED/BCBIO/RESOURCES/bcbio/anaconda/envs/dv) uses an older version:

tensorboard               2.1.0                     py3_0  
tensorflow                2.0.0           mkl_py36hef7ec59_0  
tensorflow-base           2.0.0           mkl_py36h9204916_0  
tensorflow-estimator      2.0.0              pyh2649769_0

Is there a way to specify which env to use when I run DeepVariant? Is dv _3 conda package working fine on your behalf?