search

biocore / qiime

Official QIIME 1 software repository. QIIME 2 (https://qiime2.org) has succeeded QIIME 1 as of January 2018.
GNU General Public License v2.0
285 stars 268 forks source link

Create QIIME package for anaconda #1759

Closed ElDeveloper closed 8 years ago

ElDeveloper commented 9 years ago

This would speed-up the installation process.

colinbrislawn commented 9 years ago

On RHEL 5, these commands work.

conda remove -n qiime190rc2 --all 
conda create -n qiime190rc2 python pip numpy
source activate qiime190rc2
conda install -c https://conda.binstar.org/jorge qiime

However, some tests fail. The ones relevant to my workflow are

======================================================================
FAIL: test_FastTree_supported_version (__main__.QIIMEDependencyFull)
FastTree is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/people/bris469/bin/anaconda/envs/qiime190rc2-testing/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 358, in test_FastTree_supported_version
    "'%s'." % stdout)
AssertionError: Could not find FastTree version info in usage text 'FastTree: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by FastTree)
FastTree: /lib64/libc.so.6: version 'GLIBC_2.7' not found (required by FastTree).

======================================================================
FAIL: test_blast_supported_version (__main__.QIIMEDependencyFull)
blast is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/people/bris469/bin/anaconda/envs/qiime190rc2-testing/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 470, in test_blast_supported_version
    % ('.'.join(map(str, acceptable_version)), version_string))
AssertionError: Unsupported blast version. 2.2.22 is required, but running 2.2.26.

======================================================================
FAIL: test_h5py (__main__.QIIMEDependencyFull)
h5py is installed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/people/bris469/bin/anaconda/envs/qiime190rc2-testing/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 709, in test_h5py
    "h5py is not installed. You should install this for "
AssertionError: h5py is not installed. You should install this for improved performance with large BIOM files or if working with BIOM format version 2.x files. For more information, see http://biom-format.org.

It looks like RHEL5 has a really old glibc. :confused:

Many of the biom commands will throw errors if h5py is not installed, even if it not needed. This renders the biom commands unusable.

During OTU picking, FastTree and blastall are used to align, filter, and build a .tre file. This .tre file is needed for unifrac metrics which are commonly used to estimate beta-diversity.

Assigning taxonomy with uclust works great. Some people in my lab still uses the RDP classifier through. I would like that to be included as well.

I would really like some of these 'extra' programs to be included by default, but this is a rock solid start!

Go team! Colin

ebolyen commented 9 years ago

Just ran through the Illumina overview tutorial after running these commands:

conda create -n qiime190rc2 python pip numpy 
source activate qiime190rc2
conda install -c https://conda.binstar.org/jorge qiime

It worked flawlessly! Great job!

EDIT: This was on Ubuntu 14.04

jairideout commented 9 years ago

Many of the biom commands will throw errors if h5py is not installed, even if it not needed. This renders the biom commands unusable.

@colinbrislawn that's concerning, as h5py should be an optional dependency for QIIME and BIOM. Can you please post the BIOM commands that weren't working for you?

Jorge-C commented 9 years ago

That is probably related to the h5py bugs we've been fixing in biom recently. On Jan 29, 2015 12:41 PM, "Jai Ram Rideout" notifications@github.com wrote:

Many of the biom commands will throw errors if h5py is not installed, even if it not needed. This renders the biom commands unusable.

@colinbrislawn https://github.com/colinbrislawn that's concerning, as h5py should be an optional dependency for QIIME and BIOM. Can you please post the BIOM commands that weren't working for you?

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-72090113.

jairideout commented 9 years ago

Sounds good. I thought I'd check with @colinbrislawn to see if there were other biom scripts that weren't working besides biom validate-table (would be good to catch these issues now before the BIOM/QIIME releases).

Jorge-C commented 9 years ago

:+1: On Jan 29, 2015 12:50 PM, "Jai Ram Rideout" notifications@github.com wrote:

Sounds good. I thought I'd check with @colinbrislawn https://github.com/colinbrislawn to see if there were other biom scripts that weren't working besides biom validate-table (would be good to catch these issues now before the BIOM/QIIME releases).

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-72091667.

ElDeveloper commented 9 years ago

:scream:

On (Jan-29-15|11:41), Jai Ram Rideout wrote:

Many of the biom commands will throw errors if h5py is not installed, even if it not needed. This renders the biom commands unusable.

@colinbrislawn that's concerning, as h5py should be an optional dependency for QIIME and BIOM. Can you please post the BIOM commands that weren't working for you?

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-72090113

colinbrislawn commented 9 years ago

On RHEL 6, the following commands work great!

conda create -n qiime190rc2-rhl6 python pip numpy
source activate qiime190rc2-rhl6
conda install -c https://conda.binstar.org/jorge qiime

As on RHEL 5 I get errors about the wrong glibc version. This would cause problems with fasttree and swarm. This is strange, as I've totally build trees on using fasttree and clustered with swarm. Would loading the newer gcc module solve this problem...

And of course blastall and the RDP classifier are not included in the package.

colinbrislawn commented 9 years ago

@ElDeveloper @jairideout @Jorge-C I've just tested the biom commands again. No problems! :relieved:

System information
==================
         Platform:  linux2
   Python version:  2.7.9 |Continuum Analytics, Inc.| (default, Dec 15 2014, 10:33:51)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
Python executable:  /people/bris469/bin/anaconda/envs/qiime190rc2/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.1

Dependency versions
===================
                QIIME library version:  1.9.0-rc2
                 QIIME script version:  1.9.0-rc2
      qiime-default-reference version:  0.1.1
                        NumPy version:  1.9.1
                        SciPy version:  0.15.1
                       pandas version:  0.15.2
                   matplotlib version:  1.4.2
                  biom-format version:  2.1.2
                         h5py version:  Not installed.
                         qcli version:  0.1.1
                         pyqi version:  0.3.2
                   scikit-bio version:  0.2.2
                       PyNAST version:  1.2.2
                      Emperor version:  0.9.5
                      burrito version:  0.9.0
             burrito-fillings version:  Installed.
                    sortmerna version:  Not installed.
                    sumaclust version:  Not installed.
                                gdata:  Installed.
RDP Classifier version (if installed):  Not installed.
          Java version (if installed):  1.6.0_26

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
 http://qiime.org/install/qiime_config.html
 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir:  None
      pick_otus_reference_seqs_fp:  /people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue:  all.q
      topiaryexplorer_project_dir:  None
     pynast_template_alignment_fp:  /people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.fasta
                  cluster_jobs_fp:  start_parallel_jobs.py
pynast_template_alignment_blastdb:  None
assign_taxonomy_reference_seqs_fp:  /people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue:  friendlyq
                    jobs_to_start:  1
            denoiser_min_per_core:  50
assign_taxonomy_id_to_taxonomy_fp:  /people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir:  /people/bris469/.qiime_tmp/
                     slurm_memory:  None
                      slurm_queue:  None
                      blastall_fp:  blastall
                 seconds_to_sleep:  1
(qiime190rc2)[bris469@olympus vsearch]$

These commands all work just fine. :+1:

biom summarize-table -i otu_table_w_tax.biom -o test.txt
biom summarize-table -i otu_table.txt -o test.txt2
biom convert --table-type="OTU table" -i otu_table.txt -o test.biom --to-json

Sorry to scare you guys.

jairideout commented 9 years ago

@colinbrislawn no worries, thanks for all of the testing, it's a huge help!

quantrix commented 9 years ago

This is freaking amazing! Thank you for all your hard work! I already had anaconda install. So creating the qiime env and installing in there was a breeze! I am a relative newbie to anaconda and would like to help in some way.

Also, would it be possible to install the full deploy through pip into the qiime env with the base install?

quantrix commented 9 years ago

OK, I think I asked a dumb question there as I see there are non-python dependencies in that list. My question should rather be, If I manually install the non-python additional dependencies,

1) Can I make my qiime env detect these non-pythonic dependencies, 2) How do I go about doing that?

Thanks for the hints. Once again, THANK YOU for your wonderful work!

Jorge-C commented 9 years ago

Hi @quantrix! Note that this is experimental work, so expect things to be broken... However, we deeply appreciate any testing and error reporting that you may do :) Also, thanks @colinbrislawn and @ebolyen for testing across many environments!

Regarding your questions, I think it should be possible to install the non-python dependencies manually, and they should be visible from the conda environment.

ElDeveloper commented 9 years ago

I agree, thanks everyone for the testing this. While being able to install the non-python dependencies is possible through conda, we want to make sure that the base install is working well enough that we can move on and devote more time to make a full installation possible.

Thanks for the feedback everyone!

On (Jan-29-15|22:56), Jorge Cañardo Alastuey wrote:

Hi @quantrix! Note that this is experimental work, so expect things to be broken... However, we deeply appreciate any testing and error reporting that you may do :) Also, thanks @colinbrislawn and @ebolyen for testing across many environments!

Regarding your questions, I think it should be possible to install the non-python dependencies manually, and they should be visible from the conda environment.

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-72161823

colinbrislawn commented 9 years ago

Hey @quantrix

Can I make my qiime env detect these non-pythonic dependencies.

Yes! In fact, that's what it does by default.

For example, I have usearch and blastall installed and in my path, and I have modules to load R and h5py. Whatever conda environment I am using recognizes all of them automatically. Basically, if you an can run a program before loading the conda environment, you can keep running it afterwards.

@ElDeveloper

install[ing] non-python dependencies is possible through conda

:+1: That's the holy grail!

quantrix commented 9 years ago

Thank you folks! Much appreciated! I will keep testing and see if there are any issues. For reference I installed the anaconda 2.1.0 (64 bit) - python 2.7.9 on an ubuntu 14.04 for anyone wanting to install qiime.

I tried qiime earlier with qiime-deploy. Apart from taking HOURS for the install to move forward and finally CRASH without an installation, it is a huge pain. Cannot believe how easy it was with conda. Also, FYI, I was a user of enthought for a long time before I uninstalled enthought and installed anaconda. Love, love anaconda. Am hugely motivated to learn more about anaconda.

quantrix commented 9 years ago

Hello all, I appreciate your replies and wanted to ask another question. Please bear with me. I spent the last 3 hours reading the documentation on conda online from end to end and did not find an answer.

Now, if I want to import the "qiime env" into a script that I am writing to use qiime, how do I go about doing that? Because the qiime is only an "env" which needs to be activated, can I add the code

source activate qiime under #!/usr/bin/python?

and then do an, import qiime in the script?

Also, WHERE are the qiime src code files being installed within the qiime environment and where is the original qiime src code present in this env?

I am a bit confused about this. Is this a question better addressed to conda list-serv?

Jorge-C commented 9 years ago

I do not know if it is possible to modify the environment using the shebang line (#! /usr/bin/env python), but you can definitely do

$ source activate qiime
(qiime)$ python script.py

in the command line, and your script will use whatever is installed in the qiime conda environment. Hope that helps!

Jorge-C commented 9 years ago

As for your second question, the source code of the package is in /path/to/anaconda/envs/qiime/lib/python2.7/site-packages and you can find it doing e.g.

$ python
>>> import qiime
>>> qiime.__file__
quantrix commented 9 years ago

Thanks Jorge for the quick reply!

Hmm, interesting! So in order to use my Python scripts, I need to change to the qiime env and run the scripts from there. OR, I change my default env to be the "qiime env" OR instead, maybe I can delete the qiime env and re-do an install in the root anaconda environment itself, correct?

I guess I can use the qiime env as a "test" environment and then re-install in the root environment. I want to install qiime on a linux server which is available to multiple users who are not hugely savvy with python or linux. I am trying to create scripts for routine use and trying to simplify it as much as possible...

Sorry for rambling to myself. This has been great! Thanks to all of you again!

Jorge-C commented 9 years ago

That's correct, if you install it in the root environment it will be immediately available. I guess you could also keep it in a different environment and source that environment in .bashrc to achieve a similar effect.

quantrix commented 9 years ago

Thanks Jorge. I probably should think a bit more before typing. However, it is possible to call it from the shebang line as well. You need to keep the path for your qiime python executable in mind. Here is a small script I wrote and it works.

!/home/xxxxx/anaconda/envs/qiime/bin/python

Test to run QIIME from the shebang line

import qiime

print dir(qiime)

Jorge-C commented 9 years ago

Ah, thanks @quantrix! I didn't realise that would also work :)

quantrix commented 9 years ago

Cool @Jorge-C! Anaconda and the Env model is a tad tricky for me. But I think it should be ok soon enough. But it is immensely helpful! and I LOVE it :-)

But your qiime bundle is amazingly helpful!

brooksomics commented 9 years ago

I'm relatively new to using QIIME and have had success installing and going through the Illumina tutorial by installing MacQIIME locally.

Recently I tried installing QIIME on our lab's Linux server (running a Gentoo OS), using:

conda create -n qiime190rc2-rhl6 python pip numpy
source activate qiime190rc2-rhl6
conda install -c https://conda.binstar.org/jorge qiime

Everything seems to work until the below step:

*** ERROR RAISED DURING STEP: Rarefaction plot: All metrics
Command run was:
 make_rarefaction_plots.py -i cdout//arare_max900//alpha_div_collated/ -m map.tsv -o cdout//arare_max900//alpha_rarefaction_plots/
Command returned exit status: 1
Stdout:

Stderr
Traceback (most recent call last):
  File "/home/bbrooks/miniconda/envs/qiime190rc2/bin/make_rarefaction_plots.py", line 4, in <module>
    __import__('pkg_resources').run_script('qiime==1.9.0rc2', 'make_rarefaction_plots.py')
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/setuptools-13.0.2-py2.7.egg/pkg_resources/__init__.py", line 698, in run_script

  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/setuptools-13.0.2-py2.7.egg/pkg_resources/__init__.py", line 1616, in run_script

  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/make_rarefaction_plots.py", line 229, in <module>
    main()
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/make_rarefaction_plots.py", line 219, in main
    generate_average_tables=generate_average_tables)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/qiime/make_rarefaction_plots.py", line 667, in make_averages
    metric_name, output_type)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/qiime/make_rarefaction_plots.py", line 36, in save_ave_rarefaction_plots
    plt.clf()
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 547, in clf
    gcf().clf()
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 462, in gcf
    return figure()
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 435, in figure
    **kwargs)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 47, in new_figure_manager
    return new_figure_manager_given_figure(num, thisFig)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 54, in new_figure_manager_given_figure
    canvas = FigureCanvasQTAgg(figure)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 72, in __init__
    FigureCanvasQT.__init__(self, figure)
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/backends/backend_qt4.py", line 68, in __init__
    _create_qApp()
  File "/home/bbrooks/miniconda/envs/qiime190rc2/lib/python2.7/site-packages/matplotlib/backends/backend_qt5.py", line 138, in _create_qApp
    raise RuntimeError('Invalid DISPLAY variable')
RuntimeError: Invalid DISPLAY variable

I thought this had something to do with tunneling, but when changing the tunneling flag when ssh-ing, I receive the same error.

Any thoughts?

And if this is the wrong forum, apologies. Could you please point me in the right direction?

Thank you!

Jorge-C commented 9 years ago

Glad to hear that, thanks for the report!

Re. your issue, is your server headless? This might not be related to the installation method because matplotlib doesn't particularly like that, and this is how we solved it for our travis build: https://github.com/biocore/scikit-bio/commit/6e3f6f9ea5faa00a7a214f99e7e208e4dc755f82

You can also set the backend to e.g. agg (there are some details in this SO answer)

brooksomics commented 9 years ago

Thanks for the info.

Yes the server is headless.

I'm not familiar with .yml files and am a bit lost from your instruction. Should I uninstall conda and redo my installation using the referenced .travis.yml file or is this file already installed and I just need to modify lines 14 and 15?

Jorge-C commented 9 years ago

Sorry about that, I suggest you run the following bash commands and see if that fixes the problem:

export DISPLAY=:99.0
sh -e /etc/init.d/xvfb start

Otherwise, the qiime forum might be a better place to ask. Thanks!

brooksomics commented 9 years ago

Our system's admin suggested I try ssh -Y instead of ssh -X. This allowed me to successfully finish the illumina tutorial without issue.

Thanks again for your thoughts.

Jorge-C commented 9 years ago

Thanks, that's great to hear!

2015-03-10 11:44 GMT-06:00 Bubba Brooks notifications@github.com:

Our system's admin suggested I try ssh -Y instead of ssh -X. This allowed me to successfully finish the illumina tutorial without issue.

Thanks again for your thoughts.

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-78107625.

colinbrislawn commented 9 years ago

Sorry to resurrect an old issue, but I've been doing some additional testing.

Once qiime 1.9.0 launched, were the conda packages updated?

If not, conda + pip + github works almost as well. I just ran this successfully on Mac 10.8.5 and RHEL 5.

conda info -e
conda create -n qiime191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
source activate qiime191
pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz

Edit: uses newest version of qiime.

ElDeveloper commented 9 years ago

Good to hear that.

@gregcaporaso, @jairideout, @ebolyen, @antgonza, @josenavas, @wasade, @adamrp, would any of you have an objection if @Jorge-C and I created a "biocore" account in binstar so we can officially have this and other conda packages available through www.binstar.org?

On (Apr-01-15|14:13), Colin Brislawn wrote:

Sorry to resurrect an old issue, but I've been doing some additional testing.

Once qiime 1.9.0 launched, were the conda packages updated?

If not, conda + pip + github works almost as well. I just ran this successfully on Mac 10.8.5 and RHEL 5.

conda info -e
conda -n create qiime190 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
source activate qiime190
pip install https://github.com/biocore/qiime/archive/1.9.0.tar.gz

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-88633371

antgonza commented 9 years ago

Go for it.

josenavas commented 9 years ago

:+1:

2015-04-02 5:14 GMT-07:00 Antonio Gonzalez notifications@github.com:

Go for it.

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-88880532.

Jose Navas

colinbrislawn commented 9 years ago

:+1: Awesome!

If this is something you plan on supporting in the future, I would love to see this be the recommended way to get the QIIME 'base install' in the installation tutorial. If this bundle includes some common, precomplied programs like uclust, swarm, and sortmerna, that's even better.

conda + more conda > conda + pip + github

ElDeveloper commented 9 years ago

In an ideal world we should be able to replace the qiime-deploy project with a series of conda receipes, but there's still a lot of testing to be done to verify that such plan would work, though I agree, it would be awesome!

On (Apr-02-15| 9:38), Colin Brislawn wrote:

:+1: Awesome!

If this is something you plan on supporting in the future, I would love to see this be the recommended way to get the QIIME 'base install' in the installation tutorial. If this bundle includes some common, precomplied programs like uclust, swarm, and sortmerna, that's even better.

conda + more conda > conda + pip + github

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-88968897

wasade commented 9 years ago

:+1:

On Thu, Apr 2, 2015 at 12:33 PM, Yoshiki Vázquez Baeza < notifications@github.com> wrote:

In an ideal world we should be able to replace the qiime-deploy project with a series of conda receipes, but there's still a lot of testing to be done to verify that such plan would work, though I agree, it would be awesome!

On (Apr-02-15| 9:38), Colin Brislawn wrote:

:+1: Awesome!

If this is something you plan on supporting in the future, I would love to see this be the recommended way to get the QIIME 'base install' in the installation tutorial. If this bundle includes some common, precomplied programs like uclust, swarm, and sortmerna, that's even better.

conda + more conda > conda + pip + github

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-88968897

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-89003623.

adamrp commented 9 years ago

:+1: !

On Thu, Apr 2, 2015 at 12:59 PM, Daniel McDonald notifications@github.com wrote:

:+1:

On Thu, Apr 2, 2015 at 12:33 PM, Yoshiki Vázquez Baeza < notifications@github.com> wrote:

In an ideal world we should be able to replace the qiime-deploy project with a series of conda receipes, but there's still a lot of testing to be done to verify that such plan would work, though I agree, it would be awesome!

On (Apr-02-15| 9:38), Colin Brislawn wrote:

:+1: Awesome!

If this is something you plan on supporting in the future, I would love to see this be the recommended way to get the QIIME 'base install' in the installation tutorial. If this bundle includes some common, precomplied programs like uclust, swarm, and sortmerna, that's even better.

conda + more conda > conda + pip + github

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-88968897

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-89003623.

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-89009780.

iMammal commented 9 years ago

I was able to install qiime but when i tried to run the command print_qiime_config.py -tb, it gave me the following error:-

(qiime190)[anaqib2@login-1 ~]$ print_qiime_config.py -t Traceback (most recent call last): File "/export/home/anaqib2/anaconda3/envs/qiime190/bin/print_qiime_config.py", line 4, in import('pkg_resources').run_script('qiime==1.9.0rc2', 'print_qiime_config.py') File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 723, in run_script

File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 1636, in run_script

File "/mnt/store2/home/anaqib2/anaconda3/envs/qiime190/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 835, in main() File "/mnt/store2/home/anaqib2/anaconda3/envs/qiime190/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 757, in main java_version = get_java_version() File "/mnt/store2/home/anaqib2/anaconda3/envs/qiime190/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/qiime/util.py", line 1148, in get_java_version version_line = e[0] IndexError: list index out of range

The commands I used were this:-

$ conda create -n qiime190 python pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata $ source activate qiime190 $ conda install -c https://conda.binstar.org/jorge qiime After this finished, i typed print_qiime_config.py -tb and it gave me the above error.

Any advice would be greatly appreciated.

Jorge-C commented 9 years ago

@iMammal what's the output of java -version from a command line?

iMammal commented 9 years ago

What version does it need to be? I have 1.5.0 now but that can easily be upgraded.

-bash-4.1$ java -version java version "1.5.0" gij (GNU libgcj) version 4.4.6 20120305 (Red Hat 4.4.6-4)

Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -bash-4.1$

On Fri, Apr 17, 2015 at 12:51 PM, Jorge Cañardo Alastuey < notifications@github.com> wrote:

@iMammal https://github.com/iMammal what's the output of java -version from a command line?

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-94038488.

iMammal commented 9 years ago

I tried the same set of installation commands on another account on both java 1.5.0 and after upgrading to java 1.7.0 and I got this different error:

(qiime190a)-bash-4.1$ print_qiime_config.py -tb Traceback (most recent call last): File "/export/home/clustcrilab/tools/anaconda/envs/qiime190a/bin/print_qiime_config.py", line 4, in import('pkg_resources').run_script('qiime==1.9.0rc2', 'print_qiime_config.py') File "/export/home/chukhman/.local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script self.require(requires)[0].run_script(script_name, ns) File "/export/home/chukhman/.local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script execfile(script_filename, namespace, namespace) File "/mnt/store3/clustcrilab/tools/anaconda/envs/qiime190a/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 28, in raise ImportError("%s\n%s" % (e, core_dependency_missing_msg)) ImportError: /export/home/chukhman/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString See the QIIME Installation Guide: http://qiime.org/install/install.html (qiime190a)-bash-4.1$ python -V Python 2.7.9 :: Continuum Analytics, Inc. (qiime190a)-bash-4.1$

On Fri, Apr 17, 2015 at 2:49 PM, Morris Chukhman morrisc@gmail.com wrote:

What version does it need to be? I have 1.5.0 now but that can easily be upgraded.

-bash-4.1$ java -version java version "1.5.0" gij (GNU libgcj) version 4.4.6 20120305 (Red Hat 4.4.6-4)

Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -bash-4.1$

On Fri, Apr 17, 2015 at 12:51 PM, Jorge Cañardo Alastuey < notifications@github.com> wrote:

@iMammal https://github.com/iMammal what's the output of java -version from a command line?

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-94038488.

Jorge-C commented 9 years ago

The qiime build you're using (note that it's a release candidate, we haven't updated it after the final release, so it may not have bug fixes, etc) is only compatible with UCS-4 Python. Anaconda and all its packages in linux are also UCS-4 compatible. However, your Python installation is for some reason linked to a UCS-2 libpython.so that's not from anaconda, because it should really be using UCS-4 (it can be checked using sys.maxunicode, 1114111 is UCS-4, 65535 is UCS-2).

I'd check that the LD_LIBRARY_PATH is not pointing anywhere weird, or what libraries is your python linked to ($ ld path/to/python/binary/youre/using)?

iMammal commented 9 years ago

Ok! I got it! I had to put Python 2.7.6 in my exec path to get rid of the python errors (I had 2.6.6 in there by default).

Then I got the java errors again like I had on the other account. I then switched to java 1.7.0 and I got no more errors and beautify output instead.

Thanks!

On Fri, Apr 17, 2015 at 4:09 PM, Jorge Cañardo Alastuey < notifications@github.com> wrote:

The qiime build you're using (note that it's a release candidate, we haven't updated it after the final release, so it may not have bug fixes, etc) is only compatible with UCS-4 Python. Anaconda and all its packages in linux are also UCS-4 compatible. However, your Python installation is for some reason linked to a UCS-2 libpython.so that's not from anaconda, because it should really be using UCS-4 (it can be checked using sys.maxunicode, 1114111 is UCS-4, 65535 is UCS-2).

I'd check that the LD_LIBRARY_PATH is not pointing anywhere weird, or what libraries is your python linked to ($ ld path/to/python/binary/youre/using)?

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-94075791.

cleme commented 9 years ago

@Jorge-C @brooksomics regarding the "Invalid DISPLAY variable" error, we have fixed this by creating a config file ($HOME/.config/matplotlib/matplotlibrc) for matplotlib not to use xwindows backend:

# The default backend; one of GTK GTKAgg GTKCairo GTK3Agg GTK3Cairo
# CocoaAgg MacOSX Qt4Agg Qt5Agg TkAgg WX WXAgg Agg Cairo GDK PS PDF SVG
# Template.
# You can also deploy your own backend outside of matplotlib by
# referring to the module name (which must be in the PYTHONPATH) as
# 'module://my_backend'.
backend      : Agg

and then pointing to the right place

export MATPLOTLIBRC=$HOME/.config/matplotlib/matplotlibrc

notice that the config file has to be named matplotlibrc and not .matplotlibrc, otherwise matplotlib does not get it, go figure.

surendravikram commented 9 years ago

Hi,

I am trying to install qiime using following lines

conda create -n qiime190rc2 python source activate qiime190rc2 conda install numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata conda install pip pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

but its not completing. suggest me the solution for the following error:

Thanks in advance.

[surendra@phuong Softwares]$ conda create -n qiime190rc2 python Fetching package metadata: .... Solving package specifications: . Package plan for installation in environment /home/surendra/anaconda/envs/qiime190rc2:

The following packages will be downloaded:

package build
python-2.7.10 0 13.5 MB
setuptools-17.0 py27_0 341 KB
pip-7.0.3 py27_0 1.4 MB 
------------------------------------------------------------
                                       Total:        15.2 MB

The following NEW packages will be INSTALLED:

openssl:    1.0.1k-1    
pip:        7.0.3-py27_0
python:     2.7.10-0    
readline:   6.2-2       
setuptools: 17.0-py27_0 
sqlite:     3.8.4.1-1   
system:     5.8-2       
tk:         8.5.18-0    
zlib:       1.2.8-0

Proceed ([y]/n)? y

Fetching packages ... python-2.7.10- 100% |#######################################################################################################################| Time: 0:04:40 50.38 kB/s setuptools-17. 100% |#######################################################################################################################| Time: 0:00:11 29.85 kB/s pip-7.0.3-py27 100% |#######################################################################################################################| Time: 0:00:34 42.21 kB/s Extracting packages ... [ COMPLETE ] |#########################################################################################################################################| 100% Linking packages ... [ COMPLETE ] |#########################################################################################################################################| 100%

To activate this environment, use: source activate qiime190rc2

To deactivate this environment, use: $ source deactivate

[surendra@phuong Softwares]$ source activate qiime190rc2 discarding /home/surendra/anaconda/bin from PATH prepending /home/surendra/anaconda/envs/qiime190rc2/bin to PATH (qiime190rc2)[surendra@phuong Softwares]$ conda install numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata Fetching package metadata: .... Solving package specifications: . Package plan for installation in environment /home/surendra/anaconda/envs/qiime190rc2:

The following packages will be downloaded:

package build
libpng-1.6.17 0 214 KB
freetype-2.5.2 2 2.5 MB
fontconfig-2.11.1 4 402 KB
cairo-1.12.18 4 592 KB
qt-4.8.6 3 36.3 MB
future-0.14.3 py27_0 609 KB
gdata-2.0.18 py27_0 764 KB
ipython-3.1.0 py27_1 3.4 MB
natsort-3.5.0 py27_0 26 KB
pytz-2015.4 py27_0 174 KB
pyqt-4.11.3 py27_1 3.5 MB
python-dateutil-2.4.2 py27_0 219 KB
matplotlib-1.4.3 np19py27_2 44.5 MB
pandas-0.16.1 np19py27_0 10.7 MB
scikit-bio-0.2.3 np19py27_0 947 KB 
------------------------------------------------------------
                                       Total:       104.8 MB

The following NEW packages will be INSTALLED:

cairo:           1.12.18-4        
fontconfig:      2.11.1-4         
freetype:        2.5.2-2          
future:          0.14.3-py27_0    
gdata:           2.0.18-py27_0    
ipython:         3.1.0-py27_1     
libpng:          1.6.17-0         
libxml2:         2.9.0-0          
matplotlib:      1.4.3-np19py27_2 
natsort:         3.5.0-py27_0     
numpy:           1.9.2-py27_0     
pandas:          0.16.1-np19py27_0
pixman:          0.26.2-0         
py2cairo:        1.10.0-py27_2    
pyparsing:       2.0.3-py27_0     
pyqt:            4.11.3-py27_1    
python-dateutil: 2.4.2-py27_0     
pytz:            2015.4-py27_0    
qt:              4.8.6-3          
scikit-bio:      0.2.3-np19py27_0 
scipy:           0.15.1-np19py27_0
sip:             4.16.5-py27_0    
six:             1.9.0-py27_0

Proceed ([y]/n)? y

Fetching packages ... libpng-1.6.17- 100% |#######################################################################################################################| Time: 0:00:02 85.92 kB/s freetype-2.5.2 100% |#######################################################################################################################| Time: 0:01:39 26.31 kB/s fontconfig-2.1 100% |#######################################################################################################################| Time: 0:00:07 56.27 kB/s cairo-1.12.18- 100% |#######################################################################################################################| Time: 0:00:06 88.43 kB/s qt-4.8.6-3.tar 100% |#######################################################################################################################| Time: 0:13:09 48.26 kB/s future-0.14.3- 100% |#######################################################################################################################| Time: 0:00:09 67.42 kB/s gdata-2.0.18-p 100% |#######################################################################################################################| Time: 0:00:27 28.47 kB/s ipython-3.1.0- 100% |#######################################################################################################################| Time: 0:01:18 45.12 kB/s natsort-3.5.0- 100% |#######################################################################################################################| Time: 0:00:01 16.06 kB/s pytz-2015.4-py 100% |#######################################################################################################################| Time: 0:00:07 24.75 kB/s pyqt-4.11.3-py 100% |#######################################################################################################################| Time: 0:01:15 48.79 kB/s python-dateuti 100% |#######################################################################################################################| Time: 0:00:03 58.06 kB/s matplotlib-1.4 100% |#######################################################################################################################| Time: 0:20:57 37.10 kB/s pandas-0.16.1- 100% |#######################################################################################################################| Time: 0:03:19 56.41 kB/s scikit-bio-0.2 100% |#######################################################################################################################| Time: 0:00:29 33.13 kB/s Extracting packages ... [ COMPLETE ] |#########################################################################################################################################| 100% Linking packages ... [ COMPLETE ] |#########################################################################################################################################| 100% (qiime190rc2)[surendra@phuong Softwares]$ conda install pip Fetching package metadata: .... Solving package specifications: . All requested packages already installed. packages in environment at /home/surendra/anaconda/envs/qiime190rc2:

pip 7.0.3 py27_0

(qiime190rc2)[surendra@phuong Softwares]$ pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Collecting https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Downloading https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-EzSjZV/biom-format

colinbrislawn commented 9 years ago

Hello there,

Did you read the last couple of lines of the error? ImportError: numpy must be installed prior to installing biom Looks like you have an issue with numpy. Try this: pip install numpy

Also, I think you may be using an outdated version of qiime: pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz As of today, 1.9.1 is current. Instead of that line, try running: pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz

The current release of qiime can always be found on the release page.

surendravikram commented 9 years ago

Thanks for your support it works for me after installing numpy using pip great !

colinbrislawn commented 8 years ago

Hello everyone. This issue is being addressed over in biocore/conda-recipes/issues/3. The day will come!

@Jorge-C, could you remove your 1.9.0-dev package of qiime? I'm glad that people want to install qiime through conda, but this old build makes troubleshooting tricky.

Jorge-C commented 8 years ago

Thanks for the heads up, I just deleted it. On Dec 3, 2015 2:35 PM, "Colin Brislawn" notifications@github.com wrote:

Hello everyone. This issue is being addressed over in biocore/conda-recipes#3 https://github.com/biocore/conda-recipes/issues/3. The day will come!

@Jorge-C https://github.com/Jorge-C, could you remove your 1.9.0-dev package of qiime https://anaconda.org/jorge/qiime? I'm glad that people want to install qiime through conda, but this old build makes troubleshooting tricky.

— Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-161791584.