vt fails to install

[unode]

Hi Brad,

I'm not entirely sure if I should report this issue here, on bcbio-nextgen or atks/vt.git but I'm basically following instructions after the error.

I'm trying to install bcbio-nextgen stable on a docker container using the following Dockerfile http://bpaste.net/show/231114/ . This is mostly the same included in the bcbio-nextgen repository but using stable instead of development and having granular RUN steps to save execution time in case things go wrong.

The full output log of the build of said Dockerfile, which includes the installation error at the end of the file, is: http://bpaste.net/show/231102/

Best, Renato

[chapmanb]

Renato; Thanks for the report and sorry about the issue. I'm really confused about the error message, which complains about boost, but boost isn't part of vt:

==> Cloning https://github.com/atks/vt.git
Cloning into '/.cache/Homebrew/vt--git'...
Checking connectivity... done
==> Checking out revision a7c50713594
==> make
 #include <boost/math/distributions/chi_squared.hpp>

I could not reproduce locally but bumped vt to a more recent version since there is no particular reason to be testing/debugging on that slightly older git checkout.

The best way to get a more useful error message would be to get a bash shell in your Docker container and test compilation with the verbose flag:

brew update
export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export PERL5LIB=/usr/local/lib/perl5:${PERL5LIB} && /usr/local/bin/brew install -v  --env=inherit vt

Hopefully either the update fixes the issue or the build will provide more details. Thanks for the help debugging.

[unode]

Thanks Brad,

I reran the installation and now I got something a bit more verbose. I didn't get to run the command with the verbose flag though.

This is the relevant part of the log:

==> Cloning https://github.com/atks/vt.git
Cloning into '/.cache/Homebrew/vt--git'...
Checking connectivity... done
==> Checking out revision 22894f949a
==> make
g++ -pipe -std=c++0x -O3 -ggdb -I./lib/include/ -I. -I./lib/include/htslib -I./lib/include/Rmath  -D__STDC_LIMIT_MACROS -o subset.o -c subset.cpp
g++ -pipe -std=c++0x -O3 -ggdb -I./lib/include/ -I. -I./lib/include/htslib -I./lib/include/Rmath  -D__STDC_LIMIT_MACROS -o main.o -c main.cpp
g++ -pipe -std=c++0x -O3 -ggdb -I./lib/include/ -I. -I./lib/include/htslib -I./lib/include/Rmath  -D__STDC_LIMIT_MACROS -o vt program.o hts_utils.o utils.o bam_ordered_reader.o bcf_ordered_reader.o bcf_ordered_writer.o bcf_synced_reader.o tbx_ordered_reader.o view.o index.o normalize.o merge_duplicate_variants.o variant_manip.o log_tool.o interval_tree.o genome_interval.o compute_concordance.o compute_features.o partition.o profile_indels.o profile_snps.o profile_na12878.o discover.o merge_candidate_variants.o construct_probes.o genotype.o gencode.o annotate_variants.o annotate_str.o lhmm.o chmm.o lfhmm.o rfhmm.o lhmm1.o genotyping_buffer.o lhmm_genotyping_record.o peek.o merge.o profile_mendelian.o pedigree.o concat.o decompose.o remove_overlap.o filter.o align.o subset.o estimator.o main.o lib/include/htslib/libhts.a lib/include/Rmath/libRmath.a -lz -lpthread
g++: error: estimator.o: No such file or directory
make: *** [vt] Error 1
READ THIS: https://github.com/Homebrew/homebrew/wiki/troubleshooting
If reporting this issue please do so at (not Homebrew/homebrew):
  https://github.com/chapmanb/homebrew-cbl/issues
Fatal error: local() encountered an error (return code 1) while executing 'export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`} && export PERL5LIB=/usr/local/lib/perl5:${PERL5LIB} && /usr/local/bin/brew install  --env=inherit vt'
Aborting.

Going a bit off-topic, do you think a verbose flag would be also useful in bcbio_nextgen_install.py? I had lots of other random issues while trying to install in development mode. Do you think this is something worth of an issue on the bcbio_nextgen repo?

[chapmanb]

Renato; I'm confused because I can replicate that message if I try to compile the head of vt, but the specific revision we should be checking out (22894f949a) does not have that problem. Is this from a clean image every time? Is it possible the /.cache/Homebrew/vt--git directory is messed up and needs to be removed before rerunning?

More generally, I'm confused about having lots of failures since it seems like your setup should be pretty similar to what works cleanly for you. Can you be more specific about what is failing? On my side I haven't run the automated Docker build in a couple of weeks but was hoping to finalize a new release soon and can do this and debug from my end.

Having brew use -v by default is a tradeoff -- it means a ton of compile messages which make it hard to see what is going on at a high level but would help in cases like this. It's an easy include by I don't know if it's worth it generally. I'll definitely give it some thought.

[unode]

Hi Brad,

As I mentioned before, I'm using the Dockerfile I added in the first post of this issue. What I'm basically doing is using docker build. Since all the steps up to Step 20 : RUN python bcbio_nextgen_install.py /usr/local/share/bcbio-nextgen --tooldir=/usr/local --toolplus data --nodata --nosudo -u stable have successfully completed in the past, only Step 20 gets redone and the others are skipped by using the docker container cache. As far as I understand (I'm pretty new to docker) there is no reason to expect any brew related step to be cached since it only gets pulled during step 20. The docker version I'm currently using is 0.10.0 on a Gentoo system.

As for "lots of failures", sorry to mix that with the rest. What I meant by this is that I repeatedly tried to run the original Dockerfile, the one in bcbio_nextgen repo which installs the development packages, without success. Given that the whole installation step takes quite a while, I was repeating it about once a day or once every two days. Some of these failures were related with timeouts during download hence my repeated attempts at reinstalling. In some cases, I also got other errors, but they were not reproducible which I assumed were things that got fixed in the meantime or me trying to hit a moving target. After several days of unsuccessful attempts I decided to install with the stable option instead which, although unsuccessful so far, consistently failed on the same package.

[chapmanb]

Renato; stable and development doesn't change the packages installed, only the bcbio-nextgen code. I'm not able to replicate this problem right now so am still confused, but will try to update and push a new docker image early next week and can ensure things work cleanly then. In the meantime, you could use the slightly older docker image from early this month (https://s3.amazonaws.com/bcbio_nextgen/bcbio-nextgen-docker-image.gz) if you want to get started with bcbio and docker:

gzip -dc bcbio-nextgen-docker-image.tgz | docker import - chapmanb/bcbio-nextgen-devel

Sorry about the issues and hopefully I'll have an up to date docker build early next week to finalize this.

[chapmanb]

Renato; I'm still confused as to the underlying cause of the vt build issue but verified today I could build the Docker image without any issues using the latest code and tools. This is with the standard Dockerfile in bcbio-nextgen you referenced earlier, on Ubuntu with Docker 0.9.1 or 0.10.0. I'm still confused as to the source of your error but made a new Docker image available from S3 (https://s3.amazonaws.com/bcbio_nextgen/bcbio-nextgen-docker-image.gz) so you can skip the build annoyances and use that to get the latest code. Sorry again about the problems and hope the pre-build image works for you.

[unode]

I'm already using the image you linked in the previous reply (from last week). As for vt and docker, I don't really know what is going on either. Apart from the host being a different platform (Gentoo instead of Ubuntu), everything else should be the same. The fact that I was also getting somewhat random failures, makes me trust even less the stability of docker, at least on this platform.

In case you are interested in figuring out what is going on, I think it would be best to isolate the installation of vt in a simple Dockerfile that installs homebrew and vt only. I'd do it myself, but I'm not familiar with homebrew. If not, I'm good with closing the issue.

Thanks for all the help and sorry for the confusing mess.