Suggestion for improving documentation of base Qiime install - and possible issue with all_tests.py?

[griffinp]

Hi everyone,

(Apologies if these are already known/in-progress issues or better posted on the user forum - thought they might be useful here)

Installation Steps

I have been working on setting up a completely fresh Qiime 1.9.0-rc2 install inside a virtual machine (Ubuntu precise64 Linux distribution run on a VirtualBox) that I can use for development. I found the various instructions here http://qiime.org/install/install.html somewhat useful but the layout of the document is a bit confusing. There were also numerous dependencies I had to install that aren't currently mentioned anywhere I could see. So, in case it is useful, I thought I'd make a list of the steps I took:

First fork the Qiime repository, check remote changes are up to date, clone into my working directory, make and checkout to a development branch: all as described in the CONTRIBUTING.md document here https://github.com/biocore/qiime/blob/master/CONTRIBUTING.md Then start up a new virtual machine and perform the following steps:

sudo apt-get install build-essential sudo apt-get install python-pip sudo apt-get update sudo apt-get install python-dev sudo pip install numpy sudo easy_install -U distribute sudo apt-get build-dep matplotlib

this installed freetype and png which were missing when matplotlib

was installed through the Qiime setup script

sudo pip install atlas sudo apt-get install liblapack-dev checkinstall sudo apt-get install gfortran

sudo python qiime/setup.py install

This appeared to complete the base install fine. The output of print_qiime_config.py -tb was:

System information

     Platform:  linux2

Python version: 2.7.3 (default, Dec 18 2014, 19:10:20) [GCC 4.6.3] Python executable: /usr/bin/python

QIIME default reference information

For details on what files are used as QIIME's default references, see here: https://github.com/biocore/qiime-default-reference/releases/tag/0.1.1

Dependency versions

      QIIME library version:    1.9.0-rc2
       QIIME script version:    1.9.0-rc2

qiime-default-reference version: 0.1.1 NumPy version: 1.9.1 SciPy version: 0.15.0 pandas version: 0.15.2 matplotlib version: 1.4.2 biom-format version: 2.1.2 h5py version: Not installed. qcli version: 0.1.1 pyqi version: 0.3.2 scikit-bio version: 0.2.2 PyNAST version: 1.2.2 Emperor version: 0.9.5 burrito version: 0.9.0 burrito-fillings version: Installed. sortmerna version: Not installed. sumaclust version: Not installed. gdata: Installed.

QIIME config values

For definitions of these settings and to learn how to configure QIIME, see here: http://qiime.org/install/qiime_config.html http://qiime.org/tutorials/parallel_qiime.html

                 blastmat_dir:  None
  pick_otus_reference_seqs_fp:  /usr/local/lib/python2.7/dist-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     sc_queue:  all.q
  topiaryexplorer_project_dir:  None
 pynast_template_alignment_fp:  /usr/local/lib/python2.7/dist-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.fasta
              cluster_jobs_fp:  start_parallel_jobs.py

pynast_template_alignment_blastdb: None assign_taxonomy_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta torque_queue: friendlyq jobs_to_start: 1 denoiser_min_per_core: 50 assign_taxonomy_id_to_taxonomy_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt temp_dir: /tmp/ slurm_memory: None slurm_queue: None blastall_fp: blastall seconds_to_sleep: 1

QIIME base install test results

Ran 9 tests in 0.200s

OK

Running all_tests.py

I then went ahead and ran all_tests.py.

The tests started fine, but it has now been testing qiime/tests/test_denoiser/test_cluster_util.py for over 18 hours. It doesn't appear to be hanging but is not progressing past this step. Can anybody advise whether this is normal, or possibly an issue that needs attention? I will probably try to skip this particular test as don't need to use the cluster_util.py script for my development purposes.

Thanks for your attention Pip

[jairideout]

Thanks for these detailed notes! We'll work on incorporating them into our install instructions.

Regarding the test failure, do you have the denoiser installed? This should be installed automatically as part of the base QIIME install (via pip install qiime), but it'll skip installation if you don't have ghc installed beforehand.

Can you send me the output of:

which ghc
which FlowgramAli_4frame

Thanks!

[griffinp]

I don't seem to have either of these installed. which ghc and which FlowgramAli_4frame both give no output.

I installed Qiime via 'setup.py install' rather than 'pip install qiime' - so that I could work from my development repository.

[jairideout]

You'll need to install ghc and then rerun the setup.py install command -- this should build the denoiser.

[griffinp]

I installed ghc and it appeared to build the denoiser okay:

which FlowgramAli_4frame /usr/local/bin/FlowgramAli_4frame

I'm now running all_tests.py again and it has been working on the step Testing /vagrant/qiime/tests/test_denoiser/test_cluster_util.py (the same step where it got stuck previously) for about half an hour... UPDATE: still stuck several hours later...

[jairideout]

That's strange, I'm not sure why those tests are hanging. They run in under a minute for me.

Can you run the tests in verbose mode and send the output?

python tests/test_denoiser/test_cluster_util.py -v

[griffinp]

This does seem strange. I ran that test separately as you suggested and it appeared to work fine!

vagrant@precise64:/vagrant/qiime$ python tests/test_denoiser/test_cluster_util.py -v
test_adjust_workers (__main__.TestUtils)
adjust_workers stops clients ... ok
test_check_workers (__main__.TestUtils)
check_workers checks for dead workers ... ok
test_save_send (__main__.TestUtils)
save_send reliably send data to a socket ... ok
test_setup_server (__main__.TestUtils)
setup_server opens a port and listens ... ok
test_setup_workers (__main__.TestUtils)
setup_workers starts clients ... ok
test_stop_workers (__main__.TestUtils)
stop_workers terminates all clients ... ok
test_stop_workers_on_closed_socket (__main__.TestUtils)
stop_workers terminates all clients ... ok
test_submit_jobs (__main__.TestUtils)
submit_jobs executes commands on the cluster ... ok

----------------------------------------------------------------------
Ran 8 tests in 156.096s

OK

However it still stays stuck on this test when running with all_tests.py -v and doesn't print any progress information.

[griffinp]

Any help on this issue would be much appreciated! I've been trying to get Qiime freshly up and running for a week and a half now and am still stuck at this point...

[jairideout]

Can you please send me the full output of running python tests/all_tests.py? This'll probably be too big to post on this issue, so send it to jai (dot) rideout (at) gmail (dot) com.

Are you planning to use (or further develop) the denoiser code in QIIME? If not, you can likely ignore this hanging test. Also note that there will be many other test failures if you run the full unit test suite with only a base QIIME install. If your goal is to have all tests passing, you'll need to perform a full QIIME install.

[jairideout]

Fixed in #1917.