Create QIIME package for anaconda

[ElDeveloper]

This would speed-up the installation process.

[jairideout]

:+1:, is this possible for 1.9.0?

[ElDeveloper]

I summon @Jorge-C, master of thy condaness. Do you think this will be possible? The release is planned for this Friday. I'm tied up until Thursday at noon, after that I can help with whatever is needed. I think this would be something REMARKABLE to have for 1.9.0

On (Dec-08-14| 9:46), Jai Ram Rideout wrote:

:+1:, is this possible for 1.9.0?

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-66154756

[Jorge-C]

Tachááán! I start a humongous take home final on Thursday but will try to find time before.

What dependencies would you like to include? If it's just for the base PyPI-installable package, it'd possible.

Other extra dependencies could also be made into conda packages. In fact, I think some stuff that qiime-deploy can currently install already has conda recipes in this repo. Also, it should even be possible to install R from conda in the future (@teoliphant was asking for beta testers a while ago in twitter).

[ElDeveloper]

I think we would want to only do the pypi-installable base QIIME. If this is possible that would be amazing!

It would be awesome if we could stop relying on the qiime-deploy and instead defer this task to a dedicated package manager i. e. conda. Though I think right now just getting the base-install available through conda would be more than fantastic!

[jairideout]

:+1:

[gregcaporaso]

Agree re: the base install. This would be fantastic.

[Jorge-C]

Hey, I've been able to build qiime and its base dependencies (binaries for linux64, built against numpy 1.8 are here).

Installing qiime from there kind of works. I.e. import qiime works, but something still needs to be done about scripts:

bash-4.1$ print_qiime_config.py 
Traceback (most recent call last):
  File "/Users/joca8146/anaconda/bin/print_qiime_config.py", line 4, in <module>
    __import__('pkg_resources').run_script('qiime==1.8.0-dev', 'print_qiime_config.py')
  File "/Users/joca8146/anaconda/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg/pkg_resources.py", line 517, in run_script
  File "/Users/joca8146/anaconda/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg/pkg_resources.py", line 1428, in run_script
pkg_resources.ResolutionError: No script named 'print_qiime_config.py'

Anyway, if you can test the other dependencies (e.g. create a new conda environment, get the package via conda install -c https://conda.binstar.org/jorge pck and do some testing) and report back it'd be great. Thank you!

[ElDeveloper]

Thanks @Jorge-C, this is remarkable! I figure this will probably benefit others so I'll ask here: Can I use conda in conjunction with pip, virutalenvs and virtualenvwrapper? I ask because my system uses pip+virtualenv+virtualenvwrapper so I have never really used conda and would really like to not screw up my environment (specially at this time of the academic year :P)

On (Dec-09-14|15:32), Jorge Cañardo Alastuey wrote:

Hey, I've been able to build qiime and its base dependencies (binaries for linux64, built against numpy 1.8 are here).

Installing qiime from there kind of works. I.e. import qiime works, but something still needs to be done about scripts:

bash-4.1$ print_qiime_config.py 
Traceback (most recent call last):
  File "/Users/joca8146/anaconda/bin/print_qiime_config.py", line 4, in <module>
    __import__('pkg_resources').run_script('qiime==1.8.0-dev', 'print_qiime_config.py')
  File "/Users/joca8146/anaconda/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg/pkg_resources.py", line 517, in run_script
  File "/Users/joca8146/anaconda/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg/pkg_resources.py", line 1428, in run_script
pkg_resources.ResolutionError: No script named 'print_qiime_config.py'

Anyway, if you can test the other dependencies (e.g. create a new conda environment, get the package via conda install -c https://conda.binstar.org/jorge pck and do some testing) and report back it'd be great. Thank you!

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-66379021

[Jorge-C]

I haven't used that combination but it should be possible. In linux you can install miniconda in the home folder (like we do in travis), and add ~/anaconda/bin to the path whenever you want to use conda's python. Using that python, you can create environments (conda create -n mynewcondaenv python pip) and everything inside (source activate mynewcondaenv) should be independent of everyting else. Do not use virtualenv inside conda, use conda environments instead. And don't try to install conda inside a virtualenv. pip works fine, just remember to install it in the environments so that you're don't end up using your system level pip. Maybe try it in pando or compy? Thanks @ElDeveloper!

[ElDeveloper]

@Jorge-C, this is awesome! QIIME was installed within the minute (unbelievable)!

I was able to find the problem you see and in addition I found this problem:

In [2]: from qiime import util
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-19ac84da06bb> in <module>()
----> 1 from qiime import util

/Users/yova1074/anaconda/lib/python2.7/site-packages/qiime/util.py in <module>()
     46 from cogent.cluster.procrustes import procrustes
     47 
---> 48 from skbio.util import remove_files, create_dir
     49 from skbio.format.sequences import format_fastq_record
     50 from burrito.util import ApplicationError, CommandLineApplication, FilePath

/Users/yova1074/anaconda/lib/python2.7/site-packages/skbio/__init__.py in <module>()
     59 
     60 # Add skbio.io to sys.modules to prevent cycles in our imports
---> 61 import skbio.io
     62 skbio.io  # Stop flake8 error
     63 

/Users/yova1074/anaconda/lib/python2.7/site-packages/skbio/io/__init__.py in <module>()
    305 # registry. We use import_module instead of a typical import to avoid flake8
    306 # unused import errors.
--> 307 import_module('skbio.io.clustal')
    308 import_module('skbio.io.fasta')
    309 import_module('skbio.io.fastq')

/Users/yova1074/anaconda/lib/python2.7/importlib/__init__.pyc in import_module(name, package)
     35             level += 1
     36         name = _resolve_name(name[level:], package, level)
---> 37     __import__(name)
     38     return sys.modules[name]

/Users/yova1074/anaconda/lib/python2.7/site-packages/skbio/io/clustal.py in <module>()
    121                       ClustalFormatError)
    122 from skbio.sequence import BiologicalSequence
--> 123 from skbio.alignment import Alignment
    124 
    125 

/Users/yova1074/anaconda/lib/python2.7/site-packages/skbio/alignment/__init__.py in <module>()
    233     make_identity_substitution_matrix
    234 )
--> 235 from skbio.alignment._ssw_wrapper import (
    236     StripedSmithWaterman, local_pairwise_align_ssw, AlignmentStructure)
    237 from ._exception import (SequenceCollectionError, StockholmParseError,

ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /Users/yova1074/anaconda/lib/python2.7/site-packages/skbio/alignment/_ssw_wrapper.so)

It seems like that ssw was precompiled with a different version of glibc, I checked (ldd --version) and pando has glibc 2.12 installed.

I'll try to take a look and see what's needed to solve the scripts problem.

---

Screenshot, because it was awesome:

[Jorge-C]

Fantastic! I've seen that problem before, and it happens because glibc is backward, but not forward compatible. The solution seems to be to build the packages linked to the oldest possible glibc version. I believe the anaconda guys build stuff using CentOS 5, which comes with a very old glibc.

[ElDeveloper]

That makes perfect sense, I can help you build against this version. Unless we can find an older one?

Yoshiki Vázquez-Baeza

On Dec 10, 2014, at 11:24 AM, Jorge Cañardo Alastuey notifications@github.com wrote:

Fantastic! I've seen that problem before, and it happens because glibc is backward, but not forward compatible. The solution seems to be to build the packages linked to the oldest possible glibc version. I believe the anaconda guys build stuff using CentOS 5, which comes with a very old glibc.

— Reply to this email directly or view it on GitHub.

[colinbrislawn]

So Yoshiki, Jorge, did you get the base QIIME working with conda? I'm still looking for a quick,clean way to install different versions of qiime, and this would be excellent!

Thanks! Colin

[Jorge-C]

Not yet, there are still a few unsolved issues. There should be some update in the following days though!

[colinbrislawn]

Excelent!

I just got the basic install of the qiime 1.9.0-rc1 working using anaconda without root access. Yoshiki is right, this is coolest thing ever!

Here are my notes to myself.

#####
    Setting up Anaconda
    http://davebehnke.com/using-python-anaconda-distribution.html
#####

    Download full anaconda distraction for python 2.7
    http://continuum.io/downloads#all
        (copy the link on that page and use wget or curl)

    Then
    bash Anaconda-2.1.0-Linux-x86_64.sh
    choosing any install location you like.
    Also choose to add Anaconda to your path.

    List condo environments:
    conda info -e

    Make a new environment for qiime:
    conda create -n qiime python

    Check that the new conda environment was made, then activate it.
    conda info -e
    source activate qiime

#####
    Installing qiime dependencies
#####

    I start by making sure my conda environment is loaded.
    If it is, my command prompt will start with '(qiime)'

    If not, run: 'source activate qiime'

    Install dependancies (I don't think order matters)
    conda install numpy
    conda install matplotlib
    conda install IPython
    conda install future
    conda install natsort
    conda install scipy
    conda install pandas
    conda install scikit-bio
    conda install gdata
(NOTE: these packages do not have condo packages built for them:
pyqi, cogent, qiime, qiime-default-reference, burrito, burrito-fillings, emperor, biom-format, qcli, pynast
we must install these packages using pip)   

    Install pip (within this conda environment)
    conda install pip

    attempt to install qiime-1.9.0-rc1 (into this conda environment using pip)
    pip install https://github.com/biocore/qiime/archive/1.9.0-rc1.tar.gz

    To test it run:
    print_qiime_config.py -t

    Remember that before you can use this installation, you must run:
    source activate qiime
    to activate this conda environment. 

So basically, install conda, install every qiime dependency that comes with conda, then 'pip install' like normal.

Keep me posted! :thumbsup: for continued conda integration.

[colinbrislawn]

@Jorge-C I did this under an old RHEL 5.7 system, which may be why it worked so well. I'm about to switch to a RHEL 6 box, so we'll see what happens then.

[Jorge-C]

Thanks for those notes! Let us know if they also work in the newer Red Hat :)

Our end goal is that an installation option becomes 1) setting up anaconda and 2) running conda install qiime (which would install all dependencies from binaries, no need to compile anything).

[Jorge-C]

I've just been able to get the scripts in biom-format & pyqi working from the a conda installation. Still need to get through the latest setup.py updates in qiime, then its conda package should be finally functional.

[gregcaporaso]

Thanks great news, thanks for working on this!

On Tue, Jan 20, 2015 at 7:50 PM, Jorge Cañardo Alastuey < notifications@github.com> wrote:

I've just been able to get the scripts in biom-format & pyqi working from the a conda installation. Still need to get through the latest setup.py updates in qiime, then its conda package should be finally functional.

Reply to this email directly or view it on GitHub https://github.com/biocore/qiime/issues/1759#issuecomment-70776080.

[ElDeveloper]

This is great news!! I'm excited to try this out, let me know if you need any help testing. Or building the OSX packages.

Yoshiki Vázquez-Baeza

On Jan 20, 2015, at 7:50 PM, Jorge Cañardo Alastuey notifications@github.com wrote:

I've just been able to get the scripts in biom-format & pyqi working from the a conda installation. Still need to get through the latest setup.py updates in qiime, then its conda package should be finally functional.

— Reply to this email directly or view it on GitHub.

[colinbrislawn]

This is excellent!

As a review, is the plan to make a conda package including the needed core dependencies, or to provide a tutorial?

If building or testing is needing, I'm vary willing to build and test on OSX and RHEL 5.

Colin

[Jorge-C]

Thanks guys! I'll share my notes when I clean them up so that you can try building/testing on OSX and various linux flavours. For recent linux systems, conda install -c https://conda.binstar.org/jorge qiime in a clean environment resists some smoke testing, so please report any packaging issues you may find:

(qiime_test)jorge@hplap:~$ print_qiime_config.py -t

System information
==================
         Platform:  linux2
   Python version:  2.7.9 |Continuum Analytics, Inc.| (default, Dec 15 2014, 10:33:51)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
Python executable:  /home/jorge/anaconda/envs/qiime_test/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.1

Dependency versions
===================
          QIIME library version:    1.9.0-rc2
           QIIME script version:    1.9.0-rc2
qiime-default-reference version:    0.1.1
                  NumPy version:    1.9.1
                  SciPy version:    0.15.1
                 pandas version:    0.15.2
             matplotlib version:    1.4.2
            biom-format version:    2.1.2
                   h5py version:    Not installed.
                   qcli version:    0.1.1
                   pyqi version:    0.3.2
             scikit-bio version:    0.2.2
                 PyNAST version:    1.2.2
                Emperor version:    0.9.5
                burrito version:    0.9.0
       burrito-fillings version:    Installed.
              sortmerna version:    SortMeRNA version 2.0, 29/11/2014
              sumaclust version:    SUMACLUST Version 1.0.00
                  swarm version:    Swarm 1.2.19 [Jan 21 2015 00:02:12]
                          gdata:    Installed.

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
 http://qiime.org/install/qiime_config.html
 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir:  None
      pick_otus_reference_seqs_fp:  /home/jorge/anaconda/envs/qiime_test/lib/python2.7/site-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue:  all.q
      topiaryexplorer_project_dir:  None
     pynast_template_alignment_fp:  /home/jorge/anaconda/envs/qiime_test/lib/python2.7/site-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.fasta
                  cluster_jobs_fp:  start_parallel_jobs.py
pynast_template_alignment_blastdb:  None
assign_taxonomy_reference_seqs_fp:  /home/jorge/anaconda/envs/qiime_test/lib/python2.7/site-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue:  friendlyq
                    jobs_to_start:  1
            denoiser_min_per_core:  50
assign_taxonomy_id_to_taxonomy_fp:  /home/jorge/anaconda/envs/qiime_test/lib/python2.7/site-packages/qiime_default_reference-0.1.1-py2.7.egg/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir:  /tmp/
                     slurm_memory:  None
                      slurm_queue:  None
                      blastall_fp:  blastall
                 seconds_to_sleep:  1

QIIME base install test results
===============================
.........
----------------------------------------------------------------------
Ran 9 tests in 0.028s

OK

[colinbrislawn]

Hey there,

I just tested this on a RHEL 5.27 machine. I used the following commands.

conda create -n qiime190rc2 python

source activate qiime190rc2

conda install -c https://conda.binstar.org/jorge qiime

This seemed to work great! However, when I test, I get the following error:

(qiime190rc2)[bris469@olympus bin]$ print_qiime_config.py -t
Traceback (most recent call last):
  File "/people/bris469/bin/anaconda/envs/qiime190rc2/bin/print_qiime_config.py", line 4, in <module>
    __import__('pkg_resources').run_script('qiime==1.9.0rc2', 'print_qiime_config.py')
  File "/people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/setuptools-12.0.4-py2.7.egg/pkg_resources/__init__.py", line 696, in run_script

  File "/people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/setuptools-12.0.4-py2.7.egg/pkg_resources/__init__.py", line 1614, in run_script

  File "/people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg/EGG-INFO/scripts/print_qiime_config.py", line 40, in <module>
    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /people/bris469/bin/anaconda/envs/qiime190rc2/lib/python2.7/site-packages/skbio/alignment/_ssw_wrapper.so)
See the QIIME Installation Guide: http://qiime.org/install/install.html

I did not get this error when installing with conda and pip.

Edit: This, however, totally works:

conda create -n qiime190rc2 python
source activate qiime190rc2
conda install numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
conda install pip
pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

[Jorge-C]

Thanks for testing @colinbrislawn! That's a known issue, so as you point out using conda to get most dependencies and then finishing the installation with pip is the fastest and easiest way to get it running. Note that you could also install every package at environment creation:

conda create -n qiime190rc2 python pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
source activate qiime190rc2
pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

[matt-shenton]

Hi, and thanks a lot for all this work.

(And sorry for failing at formatting here)

I just tried installing exactly as above, but I get the following (looks like it can't find numpy, somehow?)

Cheers

Matt

(qiime190rc2)[mshenton@t260 ~]$ pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Collecting https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Downloading https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz (23.6MB) 100% |################################| 23.6MB 360kB/s /home/mshenton/.local/lib/python2.7/site-packages/setuptools/dist.py:283: UserWarning: The version specified requires normalization, consider using '1.9.0rc2' instead of '1.9.0-rc2'. self.metadata.version, Requirement already satisfied (use --upgrade to upgrade): qiime==1.9.0rc2 from https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz in ./.local/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg Collecting numpy>=1.9.0 (from qiime==1.9.0rc2) Using cached numpy-1.9.1.tar.gz Running from numpy source directory. Requirement already satisfied (use --upgrade to upgrade): matplotlib>=1.1.0 in ./.local/lib/python2.7/site-packages (from qiime==1.9.0rc2) Collecting pynast==1.2.2 (from qiime==1.9.0rc2) Using cached pynast-1.2.2.tar.gz Collecting qcli<0.2.0,>=0.1.1 (from qiime==1.9.0rc2) Using cached qcli-0.1.1.tar.gz Requirement already satisfied (use --upgrade to upgrade): gdata in ./anaconda/envs/qiime190rc2/lib/python2.7/site-packages (from qiime==1.9.0rc2) Collecting biom-format<2.2.0,>=2.1.2 (from qiime==1.9.0rc2) Using cached biom-format-2.1.2.tar.gz Traceback (most recent call last): File "", line 20, in File "/tmp/pip-build-BBt1Yk/biom-format/setup.py", line 20, in raise ImportError("numpy must be installed prior to installing biom") ImportError: numpy must be installed prior to installing biom Complete output from command python setup.py egg_info: Traceback (most recent call last):

  File "<string>", line 20, in <module>

  File "/tmp/pip-build-BBt1Yk/biom-format/setup.py", line 20, in <module>

    raise ImportError("numpy must be installed prior to installing biom")

ImportError: numpy must be installed prior to installing biom

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-BBt1Yk/biom-format

(qiime190rc2)[mshenton@t260 ~]$

[ElDeveloper]

This is one of the problems with pip, you have to first install numpy before actually installing QIIME or anything that depends on NumPy. So before executing that command you have to run pip install numpy.

On (Jan-21-15|22:29), matt-shenton wrote:

Hi, and thanks a lot for all this work.

(And sorry for failing at formatting here)

I just tried installing exactly as above, but I get the following (looks like it can't find numpy, somehow?)

Cheers

Matt

(qiime190rc2)[mshenton@t260 ~]$ pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Collecting https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz Downloading https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz (23.6MB) 100% |################################| 23.6MB 360kB/s /home/mshenton/.local/lib/python2.7/site-packages/setuptools/dist.py:283: UserWarning: The version specified requires normalization, consider using '1.9.0rc2' instead of '1.9.0-rc2'. self.metadata.version, Requirement already satisfied (use --upgrade to upgrade): qiime==1.9.0rc2 from https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz in ./.local/lib/python2.7/site-packages/qiime-1.9.0rc2-py2.7.egg Collecting numpy>=1.9.0 (from qiime==1.9.0rc2) Using cached numpy-1.9.1.tar.gz Running from numpy source directory. Requirement already satisfied (use --upgrade to upgrade): matplotlib>=1.1.0 in ./.local/lib/python2.7/site-packages (from qiime==1.9.0rc2) Collecting pynast==1.2.2 (from qiime==1.9.0rc2) Using cached pynast-1.2.2.tar.gz Collecting qcli<0.2.0,>=0.1.1 (from qiime==1.9.0rc2) Using cached qcli-0.1.1.tar.gz Requirement already satisfied (use --upgrade to upgrade): gdata in ./anaconda/envs/qiime190rc2/lib/python2.7/site-packages (from qiime==1.9.0rc2) Collecting biom-format<2.2.0,>=2.1.2 (from qiime==1.9.0rc2) Using cached biom-format-2.1.2.tar.gz Traceback (most recent call last): File "", line 20, in File "/tmp/pip-build-BBt1Yk/biom-format/setup.py", line 20, in raise ImportError("numpy must be installed prior to installing biom") ImportError: numpy must be installed prior to installing biom Complete output from command python setup.py egg_info: Traceback (most recent call last):

  File "<string>", line 20, in <module>

  File "/tmp/pip-build-BBt1Yk/biom-format/setup.py", line 20, in <module>

    raise ImportError("numpy must be installed prior to installing biom")

ImportError: numpy must be installed prior to installing biom

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-BBt1Yk/biom-format

(qiime190rc2)[mshenton@t260 ~]$

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-70977352

[matt-shenton]

Hi, sorry , and thanks for the superswift reply

I didn't leav all the info- I created the conda environment as above also, including installing numpy with conda

"(qiime190rc2)[mshenton@t260 ~]$ pip install numpy Requirement already satisfied (use --upgrade to upgrade): numpy in ./.local/lib/python2.7/site-packages"

Cheers

Matt

[matt-shenton]

PS

Linux version 2.6.32-131.0.15.el6.x86_64 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Tue May 10 15:42:40 EDT 2011

[mshenton@t266 ~]$ ldd --version ldd (GNU libc) 2.12

[Jorge-C]

Hi @matt-shenton, you need to create the conda environment and install the dependencies using conda itself. If you use pip instead you may run into issues and it will take a long time to install everything. Please follow exactly the commands here if you want to try it.

Also, note that general installation and usage questions are often times better suited for qiime's google group. Thanks!

[matt-shenton]

Hi Jorge,

Thanks a lot for the reply. I'm not ashamed to admit I'm a little out of my depth here.

Actually I had first tried as above using only conda

then if I install qiime using your conda package "conda install -c https://conda.binstar.org/jorge qiime"

it appears to complete but then I get the glibc error

If I try to use pip inside the conda environment, I get the problem above that it can't find numpy

Sorry if this is in the wrong place. I have been posting on a google group thread too (although the title refers to a different problem (https://groups.google.com/forum/#!topic/qiime-forum/vlopZ0asDd8), that's what sent me here.

Anyway, thanks a lot for your help

Cheers

Matt

[Jorge-C]

No problem Matt, that's how we all learn :) Any way, don't try to use my conda package, as it's still buggy/in development, but follow Colin's suggestion instead:

# Make sure your environment is pristine if you've been trying this before
conda remove -n qiime190rc2 --all
conda create -n qiime190rc2 python pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
source activate qiime190rc2
pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

Hope that helps!

[colinbrislawn]

I really appreciate your advice @Jorge-C! I'm new to conda, so your comment is exactly what I needed.

I'll test your method on RHEL and OSX and post again.

[colinbrislawn]

On RHEL 5.7 and Mac OSX 10.8.5, the following commands work flawlessly in around 10 minutes. The only thing I did prior to this was to download and install anaconda.

# Make sure your environment is pristine if you've been trying this before
conda remove -n qiime190rc2 --all
conda create -n qiime190rc2 python pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata
source activate qiime190rc2
pip install https://github.com/biocore/qiime/archive/1.9.0-rc2.tar.gz

Thank you for your ongoing support!

[Jorge-C]

All the recipes are currently in this repo with some documentation.

Builds for linux64 with glibc >= 2.12 (can be checked which one you have using ldd --version) are available in my binstar channel: conda install -c https://conda.binstar.org/jorge qiime

Let me know how this works in clean environments. I'm particularly interested in things like the denoiser, sortmerna, etc. I haven't had time to go through the tutorial or some analyses to see how things work, and tests aren't packaged in the binaries.

Quick overview of commands used to build the binaries (done in pando):

conda install conda-build patchelf binstar
binstar login

git clone https://github.com/Jorge-C/biocore_recipes.git
cd biocore_recipes/
conda build --numpy 1.9 cogent pynast qcli pyqi emperor biom-format burrito-fillings burrito qiime-default-reference qiime
binstar upload --force ~/anaconda/conda-bld/linux-64/*tar.bz2

[ElDeveloper]

This is awesome @Jorge-C! Is there a straightforward or suggested way to create the OS X packages? When I tried the commands you listed below in, I got an error about python-sqlalchemy not being available (not sure if this is OSX specific though or if I'm missing something). Also, @gregcaporaso, @jairideout, should these be moved to a QIIME/BIOCORE account in binstar?

On (Jan-27-15|17:07), Jorge Cañardo Alastuey wrote:

All the recipes are currently in this repo with some documentation.

Builds for linux64 with glibc >= 2.12 (can be checked which one you have using ldd --version) are available in my binstar channel: conda install -c https://conda.binstar.org/jorge qiime

Let me know how this works in clean environments. I'm particularly interested in things like the denoiser, sortmerna, etc. I haven't had time to go through the tutorial or some analyses to see how things work, and tests aren't packaged in the binaries.

Quick overview of commands used to build the binaries (done in pando):

conda install conda-build patchelf binstar
binstar login

git clone https://github.com/Jorge-C/biocore_recipes.git
cd biocore_recipes/
conda build --numpy 1.9 cogent pynast qcli pyqi emperor biom-format burrito-fillings burrito qiime-default-reference qiime
binstar upload --force ~/anaconda/conda-bld/linux-64/*tar.bz2

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-71762781

[Jorge-C]

I think they shouldn't be moved till they are verified to "work" :)

Re. your question, the steps should be the same. I'll investigate why that package isn't available in OSX. If you have the error message available, can you please post it?

[ElDeveloper]

I agree, we have to do more testing.

The full output that I get when I run conda build ... is here: https://gist.github.com/ElDeveloper/96cffb9525cc2a5f78ef

The relevant error message is this:

BUILD END: cogent-1.5.3-np19py27_0
TEST START: cogent-1.5.3-np19py27_0
Fetching package metadata: .....
Solving package specifications: 
Error: Could not find some dependencies for cogent 1.5.3 np19py27_0:
mysql-python

On (Jan-28-15|11:18), Jorge Cañardo Alastuey wrote:

I think they shouldn't be moved till they are verified to "work" :)

Re. your question, the steps should be the same. I'll investigate why that package isn't available in OSX. If you have the error message available, can you please post it?

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-71896879

[Jorge-C]

Thanks Yoshiki! For some reason that package is not available in OSX from anaconda:

bash-4.1$ conda search --platform osx-64 mysql-python
Fetching package metadata: ...
bash-4.1$ conda search --platform linux-64 mysql-python
Fetching package metadata: ...
mysql-python              *  1.2.3                    py27_0  defaults        

At any rate, that is not necessary unless using the Ensembl querying capabilities from pycogent, so it's in fact an extra dependency. Do you know if QIIME uses that?

[ElDeveloper]

No problem. I am sure it is not used anywhere in QIIME. @gregcaporaso or @antgonza do can you guys confirm this?

On (Jan-28-15|13:51), Jorge Cañardo Alastuey wrote:

Thanks Yoshiki! For some reason that package is not available in OSX from anaconda:

bash-4.1$ conda search --platform osx-64 mysql-python
Fetching package metadata: ...
bash-4.1$ conda search --platform linux-64 mysql-python
Fetching package metadata: ...
mysql-python              *  1.2.3                    py27_0  defaults        

At any rate, that is not necessary unless using the Ensembl querying capabilities from pycogent, so it's in fact an extra dependency. Do you know if QIIME uses that?

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-71923999

[Jorge-C]

Cool, I just asked if there're any reasons for it missing in anaconda.

[Jorge-C]

In the mean time, feel free to pull from my repo and try once again (I've removed that dependency from cogent, and pushed new binaries to my binstar channel).

[ElDeveloper]

Thanks for the help! I tried again, and found the following error:

Fetching packages ...
openmpi-1.6.3- 100% |################################| Time: 0:00:03   1.99 MB/s
six-1.9.0-py27 100% |################################| Time: 0:00:00 246.65 kB/s
mpi4py-1.3-py2 100% |################################| Time: 0:00:01 304.47 kB/s
matplotlib-1.4 100% |################################| Time: 0:00:11   3.82 MB/s
cogent-1.5.3-n 100% |################################| Time: 0:00:00  85.46 MB/s
Extracting packages ...
[      COMPLETE      ] |##################################################| 100%
Linking packages ...
[matplotlib-1.4.2-np19py27_0 /Users/yoshikivazquezbaeza/miniconda/pkgs 1] ||  94[      COMPLETE      ] |##################################################| 100%
===== testing package: cogent-1.5.3-np19py27_0 =====
import: u'cogent'
import: u'cogent.align'
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    opal_init:startup:internal-failure
But I couldn't open the help file:
    /opt/anaconda1anaconda2anaconda3/share/openmpi/help-opal-runtime.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
[debugatron-2000.local:15741] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:startup:internal-failure
But I couldn't open the help file:
    /opt/anaconda1anaconda2anaconda3/share/openmpi/help-mpi-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
[debugatron-2000.local:15741] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!
TESTS FAILED: cogent-1.5.3-np19py27_0

Full output: https://gist.github.com/ElDeveloper/4f8cb2dbc4a19dd6a268

I'm still in Boulder, so maybe we could meet some time and figure this out together.

On (Jan-28-15|14:04), Jorge Cañardo Alastuey wrote:

In the mean time, feel free to pull from my repo and try once again (I've removed that dependency from cogent, and pushed new binaries to my binstar channel).

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-71926252

[Jorge-C]

OpenMPI errors? That's kind of funny hehe.

Sounds perfect, I'll send you an email to get together.

[ElDeveloper]

Great, we'll update the rest of the group through this issue.

On (Jan-28-15|14:32), Jorge Cañardo Alastuey wrote:

OpenMPI errors? That's kind of funny hehe.

Sounds perfect, I'll send you an email to get together.

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-71930978

[ElDeveloper]

@Jorge-C and I just met, we were able to get the recipes to work in OS X. All I had to do to compile and build the binaries was use the recipes that @Jorge-C created. The end result is being able to get a base install of QIIME within one minute by doing:

#### WARNING, I HAVEN'T TESTED THIS IN A WHILE AND MAY NO LONGER BE
#### WORKING, -y January 23, 2016
# EXCLUSIVELY FOR OSX
conda install -c https://conda.binstar.org/yoshiki qiime

# EXCLUSIVELY FOR LINUX GLIBC >= 2.12
conda install -c https://conda.binstar.org/jorge qiime

If others can test and let us know if there are problems, that would be awesome!

@gregcaporaso, @jairideout, @antgonza do you guys know if we really need mysql-python or mpi4py for QIIME to work correctly? AFAIK no, but I wanted to confirm this.

---

Obligatory animated gif (note that I had a problem because my qiime config had a deprecated value):

[jairideout]

This is awesome! I'll try to find some time today to test on my end (OS X), though I need to prioritize the remaining to-do items for the release. @ebolyen can you test this out on Ubuntu?

@gregcaporaso, @jairideout, @antgonza do you guys know if we really need mysql-python or mpi4py for QIIME to work correctly? AFAIK no, but I wanted to confirm this.

AFAIK QIIME doesn't require these dependencies, though I seem to remember PyCogent throwing warnings if they weren't present (which would show up when you'd run certain QIIME scripts). @gregcaporaso @antgonza do you have any more details about these dependencies?

[ElDeveloper]

Great, thanks for the information! And thanks for testing.

For others who would like to test but don't have miniconda installed, the installation process is very straightforward and shouldn't interfere with your virtual environments (if you have any).

On (Jan-29-15| 8:55), Jai Ram Rideout wrote:

This is awesome! I'll try to find some time today to test on my end (OS X), though I need to prioritize the remaining to-do items for the release. @ebolyen can you test this out on Ubuntu?

@gregcaporaso, @jairideout, @antgonza do you guys know if we really need mysql-python or mpi4py for QIIME to work correctly? AFAIK no, but I wanted to confirm this.

AFAIK QIIME doesn't require these dependencies, though I seem to remember PyCogent throwing warnings if they weren't present (which would show up when you'd run certain QIIME scripts). @gregcaporaso @antgonza do you have any more details about these dependencies?

---

Reply to this email directly or view it on GitHub: https://github.com/biocore/qiime/issues/1759#issuecomment-72060698

[ebolyen]

@jairideout yup! I'll report back with my results.

[colinbrislawn]

Hey everybody,

On Mac OSX 10.8.5, these commands work great!

# Make sure your environment is pristine if you've been trying this before
conda remove -n qiime190rc2 --all
conda create -n qiime190rc2 python pip numpy 
source activate qiime190rc2
conda install -c https://conda.binstar.org/yoshiki qiime

Does this include executables like blastall and the RDP classifier? That would be nice...

[Jorge-C]

The extra dependencies (i.e. what qiime-deploy installs right now) will be the next step!

Let us know if your analyses also work well with that installation (and maybe make sure that your $PATH and $PYTHONPATH aren't pointing to other places where you might have qiime and its dependencies installed).

Thanks for trying, it's great to hear that!!