search

snayfach / MIDAS

An integrated pipeline for estimating strain-level genomic variation from metagenomic data
http://dx.doi.org/10.1101/gr.201863.115
GNU General Public License v3.0
119 stars 52 forks source link

Cannot use MIDAS with python2 conda environment #69

Closed jolespin closed 6 years ago

jolespin commented 6 years ago

I use Python 3.6.2 for my main environment and created a new conda environment for Python 2.7.13 in which I installed MIDAS. I am trying to run MIDAS but I am encountering the following errors regarding libbz2.so.1.0, gzopen64 and bowtie2-align.

Do any of these look familiar?

Added this to my bash profile:

# Python2 Alias (I also have an environment called `python2`)
alias python2='/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/python'

# MIDAS
export PYTHONPATH=$PYTHONPATH:/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS
export PATH=$PATH:/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/scripts
export MIDAS_DB=/usr/local/devel/ANNOTATION/jespinoz/db/midas_db_v1.2

I ran this command after activating my python2 environment:

#!/bin/bash
echo S-1504-86.B
python run_midas.py species /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/S-1504-86.B_midas -1 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_1_sub-0.25.fastq.gz -2 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_2_sub-0.25.fastq.gz -t 2
python run_midas.py genes /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/S-1504-86.B_midas -1 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_1_sub-0.25.fastq.gz -2 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_2_sub-0.25.fastq.gz -t 2
python run_midas.py snps /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/S-1504-86.B_midas -1 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_1_sub-0.25.fastq.gz -2 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_2_sub-0.25.fastq.gz -t 2

Encountered this error:

Error encountered executing:
/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/bowtie2 --no-unal -x /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/S-1504-86.B_midas/snps/temp/genomes --very-sensitive --threads 2 -q -1 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_1_sub-0.25.fastq.gz -2 /usr/local/projdata/0497/projects/CariesBiome/metagenomics/run1/subsample-0.25_assembly/fastq_files/S-1504-86.B_RD1_kneaddata_paired_2_sub-0.25.fastq.gz | /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/samtools view -b - --threads 2 | /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/samtools sort - --threads 2 -o /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/S-1504-86.B_midas/snps/temp/genomes.bam

Error message:
/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/samtools: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory
/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/samtools: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory
/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/bowtie2-align-s: symbol lookup error: /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux/bowtie2-align-s: undefined symbol: gzopen64
(ERR): bowtie2-align exited with value 127
snayfach commented 6 years ago

Hi Josh,

It looks like the issue is with the precompiled samtools (v1.4) and/or bowtie2 (v2.3.2) binaries included with the software. To fix this, I would suggest downloading and compiling these programs on your own system. Once complete, copy them to MIDAS/bin/Linux. You can see the binaries MIDAS needs here: https://github.com/snayfach/MIDAS/tree/master/bin/Linux. Let me know if this works for you. I will probably come up with a better solution in the future.

Best, Stephen

jolespin commented 6 years ago

So far it's working with the exception of one error (I believe when calling snps). I replaced the binaries with symbolic links to the samtools and bowtie2 already installed on the machine: 

cd /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux
ln -s /usr/local/bin/samtools samtools
ln -s /usr/local/bin/bowtie2 bowtie2
ln -s /usr/local/bin/bowtie2-align-l bowtie2-align-l
ln -s /usr/local/bin/bowtie2-align-s bowtie2-align-s
ln -s /usr/local/bin/bowtie2-build bowtie2-build
ln -s /usr/local/bin/bowtie2-build-s bowtie2-build-s
ln -s /usr/local/bin/bowtie2-build-l bowtie2-build-l

This is the error:

Error: Encountered internal Bowtie 2 exception (#1)
Command: bowtie2-build --wrapper basic-0 --threads 2 /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/2028.2_midas/snps/temp/genomes.fa /usr/local/projdata/0497/projects/CariesBiome/jespinoz/pt_II/metagenome/midas_output/2028.2_midas/snps/temp/genomes
snayfach commented 6 years ago

Glad to hear it! Just double check that they are the correct versions if you haven't already.

On Thu, Sep 14, 2017 at 12:53 PM, Josh L. Espinoza <notifications@github.com

wrote:

So far it's working. I replaced the binaries with symbolic links to the samtools and bowtie2 already installed on the machine:

cd /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux ln -s /usr/local/bin/samtools samtools ln -s /usr/local/bin/bowtie2 bowtie2 ln -s /usr/local/bin/bowtie2-align-l bowtie2-align-l ln -s /usr/local/bin/bowtie2-align-s bowtie2-align-s ln -s /usr/local/bin/bowtie2-build bowtie2-build ln -s /usr/local/bin/bowtie2-build-s bowtie2-build-s ln -s /usr/local/bin/bowtie2-build-l bowtie2-build-l

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/snayfach/MIDAS/issues/69#issuecomment-329590790, or mute the thread https://github.com/notifications/unsubscribe-auth/ACAbrXGAqQ7wHKGcggOmLoVXvOYzg0sfks5siYRBgaJpZM4PX4Is .

jolespin commented 6 years ago

bowtie2 I'm using is an older 2.2.5 version b/c I was getting the following error when I tried using the newest version: 

/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/compile_binaries/bowtie2-2.3.2/bowtie2-align-s: error while loading shared libraries: libtbbmalloc_proxy.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
snayfach commented 6 years ago

If you run the test script you will find out if your binaries will work or not with MIDAS

On Thu, Sep 14, 2017 at 1:10 PM, Josh L. Espinoza notifications@github.com wrote:

bowtie2 is an older 2.2.5 version b/c I was getting the following error when I tried using the newest version:

/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/compile_binaries/bowtie2-2.3.2/bowtie2-align-s: error while loading shared libraries: libtbbmalloc_proxy.so.2: cannot open shared object file: No such file or directory (ERR): Description of arguments failed!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/snayfach/MIDAS/issues/69#issuecomment-329595020, or mute the thread https://github.com/notifications/unsubscribe-auth/ACAbrVT9mCHcalMFo9lzCNkJ7vTx9wQqks5siYgzgaJpZM4PX4Is .

jolespin commented 6 years ago

Ended up installing bowtie2 v2.3.0 and samtools v1.5 with conda and it worked:

conda install bowtie2 --yes conda install samtools --yes

Updated my symbolic links and it works. 

cd /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/lib/python2.7/site-packages/MIDAS/bin/Linux
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/samtools samtools
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2 bowtie2
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2-align-l bowtie2-align-l
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2-align-s bowtie2-align-s
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2-build bowtie2-build
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2-build-s bowtie2-build-s
ln -s /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/python2/bin/bowtie2-build-l bowtie2-build-l