Docker HDF5 bug via "import h5py"

[CRMacPherson]

Great work on the Docker image, its nice to see examples!

I've run into a "file not found" type error. Full traceback from the mini_test script below. I thought you might not have updated the docker image for HDF5, so I tried (re)installing the hdf5 library, in the container, as per the instructions on your site but no luck.

"""

Generating an EMPTY contigs database ...

Traceback (most recent call last): File "/usr/local/bin/anvi-gen-contigs-database", line 7, in import anvio.dbops as dbops File "/usr/local/lib/python2.7/dist-packages/anvio/dbops.py", line 28, in import anvio.auxiliarydataops as auxiliarydataops File "/usr/local/lib/python2.7/dist-packages/anvio/auxiliarydataops.py", line 4, in import h5py File "/usr/local/lib/python2.7/dist-packages/h5py/init.py", line 13, in from . import _errors ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory """

[meren]

Crap :/

This is my fault. I am sorry. I went back to my server logs, and it seems I've been testing the docker image for 1.2.0 by running this command:

docker run --rm -it meren/anvio:1.1.0

So I never tested 1.2.0 before uploading it :(

I will be very busy today, and I asked a friend to take a look. If no one responds to this, I will try to resolve it as soon as I can. Meanwhile please try the docker image for 1.1.0 version for now.

[CRMacPherson]

OK, no problem, thanks for the quick reply. I can confirm the error is not repeatable in tag 1.1.0.

A new error in 1.1.0: running "anvi-profile -i name.bam --list-contigs | head" says that the -c option is required. The documentation for anvi-profile says either -i or -c should be specified. The tutorial (http://merenlab.org/2015/05/02/anvio-tutorial/) says that the minimal command requires both -i and -c, "anvi-profile -i X.bam -c contigs.db".

I'll create an official issue once the 1.2.0 image is stable.

[meren]

This is what happens when cute projects really start being used :/ If you bear with us, we will get all of them sorted out. Sorry about these embarrassing hiccups, and thanks for your patience.

[meren]

I found the problem with the Docker image. I will upload an updated docker image in 10-15 minutes.

[meren]

Hi @CRMacPherson,

I just pushed a new 1.2.0 that should be working (the image id is 110b749e53d1). I'd be very happy if you can confirm whether it works. Thanks a lot.

[CRMacPherson]

@meren Hey, can confirm the new layer 110b749e53d1 pulled successfully and the mini_test script ran without error. Nicely done and thanks for the prompt response.

Maybe you can mention what the problem was for all those people not using Docker?

[meren]

Thanks!

In order to save space in the resulting docker image, we had this command that removes all packages that are necessary for building stuff, but are not used during runtime:

apt-get remove -y \
        binutils \
        build-essential \
        g++ \
        g++-4.8 \
        gcc \
        gcc-4.8 \
        make \
        patch \
        $(dpkg --list | grep "^ii" | grep "\-dev" | awk '{print $2}' | xargs) \

The very last parameter in this one-liner removes every package that ends with -dev to get rid of all the development packages once we're done with compiling things in the image. Along with everything else it was removing libhdf5-dev from the system, and as a result h5py was complaining about the missing shared object (libhdf5.so.7 --which I am not sure whether it should be shipped in a *-dev package, but I will not start a war with anyone over this).

As a quick resolution I removed the line from our Dockerfile, so all *-dev packages stay in the image (which increased the size about 80Mb. For which I will do a better job later).

Thanks for your patience.

[CRMacPherson]

Great thanks again!

Also Docker related,

Perhaps mention in the docs how to connect the host's browser to the interactive server...

Assuming the use of a the Docker Toolbox on Mac/Windows:

1. After starting a VM with the quick start terminal take note of the IPAddress. It will look something like this "docker is configured to use the default machine with IP 192.168.99.100"
2. Start the docker container with docker run -p 8080:8080 -it meren/anvio:1.2.0
3. Change directory to "anvi-tests"
4. Run "./run_mini_test.sh". All tests will run and at the end it will say 'The server is now listening the port number "8080"'.
5. Finally, go to your Chrome browser and type the VM's IPAddress that you noted in (1) and the port 8080. 5.1. For this example, "192.168.99.100:8080"

[meren]

This is a good idea. It forced me to install Docker Tools on my Mac.

I will test the performance, and add a section for people who would like to do it on OSX or Windows.

Thank you very much!

[meren]

I updated the post, @CRMacPherson, thanks again for the suggestion.