search

google / deepvariant

DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.
BSD 3-Clause "New" or "Revised" License
3.23k stars 727 forks source link

Compiled from source, tests pass but "make examples" does not run with test data. #199

Closed moscow25 closed 5 years ago

moscow25 commented 5 years ago

Hi! I compiled from source on Ubuntu 14.4.

Tests pass, including ./bazel-bin/deepvariant/make_examples_test.

However, running as simple make_examples with downloaded example data fails with confusing (for me) error in realign script. Same script worked fine when I ran pre-compiled binary from the Docker container 🤷‍♂️

input="./quickstart-testdata"

./bazel-bin/deepvariant/make_examples \
  --ref=$input/ucsc.hg19.chr20.unittest.fasta \
  --reads=$input/NA12878_S1.chr20.10_10p1mb.bam \
  --regions "chr20:10,000,000-10,010,000" \
  --examples examples.tfrecord@1.gz \
  --mode calling \
  --logging_every_n_candidates 10 \
  --realign_reads
./make_examples_demo.sh 
2019-07-16 17:49:02.877175: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:02.877284 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I0716 17:49:03.117142 139897470359360 make_examples.py:1110] Preparing inputs
2019-07-16 17:49:03.117644: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:03.117749 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I0716 17:49:03.118745 139897470359360 make_examples.py:1034] Common contigs are [u'chr20']
I0716 17:49:03.120177 139897470359360 make_examples.py:1116] Writing examples to examples.tfrecord-00000-of-00001.gz
2019-07-16 17:49:03.121118: I third_party/nucleus/io/sam_reader.cc:600] Setting HTS_OPT_BLOCK_SIZE to 134217728
2019-07-16 17:49:03.124279: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:03.124422 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1235, in <module>
    tf.app.run()
  File "/home/nyakovenko/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1225, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1127, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 849, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 889, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 606, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 232, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 84, in _candidates_from_reads
    region, expanded_region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 152, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance
akolesnikov commented 5 years ago

What Python version do you have installed?

moscow25 commented 5 years ago

Thanks @akolesnikov for responding!

I'm running Python Python 2.7.15+. This was on a fresh Ubuntu 14.4 install, using your default settings in DeepVariant v0.8/master and running ./build-prereq.sh followed by ./build_and_test.sh

What I'm confused by is that bazel-bin/deepvariant/make_examples_test runs and everything passes...

$ bazel-bin/deepvariant/make_examples_test
Running tests under Python 2.7.15: /usr/bin/python
...
.
----------------------------------------------------------------------
Ran 101 tests in 6.501s

OK
moscow25 commented 5 years ago

@akolesnikov do you have any suggestions for other commands to try? Roll back to previous version? Different parameters passed to the function? Other ways to run the Python code? I am not very familair with Bazel to be honest. Just confusing how the test pass, and same exact command works when running from pulled Docker container...

Happy to provide more details. Any ideas? Thanks!

pichuan commented 5 years ago

Our build and test was tested on Ubuntu 16.4. I can do a quick test run on Ubuntu 14.4. Can you remind us what version of bazel you're using?

moscow25 commented 5 years ago

[bazel release 0.21.0] -- should I try a more recent version?

I believe this was prescribed by default settings.sh. https://github.com/google/deepvariant/blob/r0.8/settings.sh

Is there a good review stable release from which to try? Sorry. Really at a loss here, as tests pass.

moscow25 commented 5 years ago

I've also tried on Ubuntu 18.04 and getting similar problems. Have you tried these test on systems other than 16.04, or should I really try to downgrade to that exact version? Thanks!

Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-54-generic x86_64)

pichuan commented 5 years ago

We're consulting with our teammate @ThomasColthurst to see if he has anything to add here about your error message.

In the past, I did try building with both Ubuntu 14 and 18 and didn't encounter such issue, but I have not done so recently. I'll try soon.

And bazel - I think 0.21 is the right version we're using.

pichuan commented 5 years ago

Quick answer: Can you try ./build_release_binaries.sh instead of ./build_and_test.sh in your steps above? Then it should work. The issue you're encountering has nothing to do with Ubuntu 18.

More details:

Earlier today, @akolesnikov and I were just wondering why our internal tests didn't capture this. We have daily tests that run scripts like this: https://github.com/google/deepvariant/blob/r0.8/scripts/run_wes_case_study_binaries.sh

After checking what that script was doing, I found out that if you run scripts/run_wes_case_study_binaries.sh, it'll work on both Ubuntu 16 and 18.

We'll fix build_and_test.sh in the next release. Thanks for reporting! If using build_release_binaries.sh still doesn't work for you, feel free to reopen.

moscow25 commented 5 years ago

Thanks @pichuan and @akolesnikov -- indeed you're right ./build_release_binaries.sh solves this.

Not sure what part of the documentation I missed, but indeed it's working now. Thanks!

pichuan commented 5 years ago

@moscow25 I don't think you missed anything; we just forgot to keep scripts consistent. Thanks for your report and we'll make sure to fix it in the future!

animesh commented 5 years ago

yes @pichuan scripts/run_wes_case_study_binaries.sh does complete with those warnings, just wondering about why the output is binary when the postprocess command is invoked with tfrecord in name examples.tfrecord.zip

python ./bazel-bin/deepvariant/postprocess_variants.zip --ref quickstart-testdata/ucsc.hg19.chr20.unittest.fasta --infile quickstart-output/examples.tfrecord.cvo.gz --outfile quickstart-output/examples.tfrecord.vcf

than without examples.zip

python ./bazel-bin/deepvariant/postprocess_variants.zip --ref quickstart-testdata/ucsc.hg19.chr20.unittest.fasta --infile quickstart-output/examples.tfrecord.cvo.gz --outfile quickstart-output/examples.vcf

and what does that binary file represent?

pichuan commented 5 years ago

Hi @animesh I'm not sure you meant exactly. Do you mean that in two different places of our documentation, we used quickstart-output/examples.tfrecord.vcf and quickstart-output/examples.vcf? I did a quick search on our Quick Start, but couldn't find it. (BTW, I noticed that Quick Start mentioned "docker_entrypoint.py", which has been renamed to https://github.com/google/deepvariant/blob/r0.8/scripts/run_deepvariant.py . In case you're looking for that)

If you are asking whether the two commands you posted are different. It seems to me the only difference is the output file name is different. So, first run should give you a VCF file with name examples.tfrecord.vcf, second should give you examples.vcf, but the content should be exactly the same.

If that's not what you're asking, please feel free to post again.

kokyriakidis commented 4 years ago

Hi @pichuan !

Do you have a solution or a clue about this issue?

https://github.com/bcbio/bcbio-nextgen/issues/3048

pichuan commented 4 years ago

Hi @kokyriakidis , in this thread (when it was still r0.8), running ././build-prereq.sh && ./build_release_binaries.sh fixed the original user's question. In the latest release (r0.9), it should work even if you run build_and_test.sh instead of build_release_binaries.sh.

The main trick was this fix_zip_rule that makes sure the symbolic links are correct: https://github.com/google/deepvariant/blob/97cd861800ccb43d750f392b518e99d514adddd8/build_release_binaries.sh#L41

In the issue that you reported in bcbio/bcbio-nextgen#3048, there is one more layer because you're using bioconda.

I'm surprised that your issue was actually with v0.9.0. Given that we actually fixed build_and_test.sh in the latest version.

@kokyriakidis Two questions for you:

  1. With bioconda, were you able to run with v0.8.0 like @chapmanb suggested?
  2. Did you actually try building DeepVariant binaries on your own? If you tried, did that work for you or did you get the same error?