search

BinPro / CONCOCT

Clustering cONtigs with COverage and ComposiTion
Other
124 stars 48 forks source link

No module named vbgmm #181

Closed NielQC closed 6 years ago

NielQC commented 6 years ago

Hi all,

I have installed all the dependencies, included the gsl library. However COCNOCT is not able to run cause it can't find the 'vbgmm' module. I have read all the threads treating this problem, but none solved the problem. The installation command runs smoothly as far as I know. Here is the output of sudo python ./setup install :

running install running bdist_egg running egg_info creating concoct.egg-info writing requirements to concoct.egg-info/requires.txt writing concoct.egg-info/PKG-INFO writing top-level names to concoct.egg-info/top_level.txt writing dependency_links to concoct.egg-info/dependency_links.txt writing entry points to concoct.egg-info/entry_points.txt writing manifest file 'concoct.egg-info/SOURCES.txt' reading manifest file 'concoct.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'concoct.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/concoct copying concoct/input.py -> build/lib.linux-x86_64-2.7/concoct copying concoct/parser.py -> build/lib.linux-x86_64-2.7/concoct copying concoct/transform.py -> build/lib.linux-x86_64-2.7/concoct copying concoct/init.py -> build/lib.linux-x86_64-2.7/concoct copying concoct/output.py -> build/lib.linux-x86_64-2.7/concoct copying concoct/cluster.py -> build/lib.linux-x86_64-2.7/concoct running build_ext building 'vbgmm' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/c-concoct x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Ic-concoct -I/usr/include/python2.7 -c c-concoct/vbgmmmodule.c -o build/temp.linux-x86_64-2.7/c-concoct/vbgmmmodule.o c-concoct/vbgmmmodule.c: In function ‘driver’: c-concoct/vbgmmmodule.c:121:28: warning: assignment discards ‘const’ qualifier from pointer target type [enabled by default] ptBestCluster->szCOutFile = szFileStub; ^ c-concoct/vbgmmmodule.c:147:5: warning: implicit declaration of function ‘gsl_blas_dgemm’ [-Wimplicit-function-declaration] gsl_blas_dgemm (CblasNoTrans,CblasNoTrans,1.0,tData.ptTMatrix,ptBestCluster->aptSigma[k],0.0,ptTemp); ^ c-concoct/vbgmmmodule.c: In function ‘performMStep’: c-concoct/vbgmmmodule.c:661:17: warning: unused variable ‘ptSigmaMatrix’ [-Wunused-variable] gsl_matrix* ptSigmaMatrix = gsl_matrix_alloc(nD,nD); ^ c-concoct/vbgmmmodule.c: In function ‘calcVBL’: c-concoct/vbgmmmodule.c:1393:7: warning: implicit declaration of function ‘gsl_blas_dsymv’ [-Wimplicit-function-declaration] gsl_blas_dsymv (CblasLower, 1.0, ptCluster->aptSigma[k], ptDiff, 0.0, ptR); ^ c-concoct/vbgmmmodule.c:1395:7: warning: implicit declaration of function ‘gsl_blas_ddot’ [-Wimplicit-function-declaration] gsl_blas_ddot (ptDiff, ptR, &dT2); ^ c-concoct/vbgmmmodule.c: In function ‘writeTMeans’: c-concoct/vbgmmmodule.c:1636:7: warning: implicit declaration of function ‘gsl_blas_dgemv’ [-Wimplicit-function-declaration] gsl_blas_dgemv (CblasNoTrans, 1.0,ptData->ptTMatrix,ptVector,0.0, ptTVector); ^ c-concoct/vbgmmmodule.c: In function ‘runRThreads’: c-concoct/vbgmmmodule.c:1683:15: warning: variable ‘iret’ set but not used [-Wunused-but-set-variable] int iret[N_RTHREADS]; ^ x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/c-concoct/vbgmmmodule.o -lgsl -lgslcblas -o build/lib.linux-x86_64-2.7/vbgmm.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/input.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/parser.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/transform.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/init.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/output.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/concoct/cluster.py -> build/bdist.linux-x86_64/egg/concoct copying build/lib.linux-x86_64-2.7/vbgmm.so -> build/bdist.linux-x86_64/egg byte-compiling build/bdist.linux-x86_64/egg/concoct/input.py to input.pyc byte-compiling build/bdist.linux-x86_64/egg/concoct/parser.py to parser.pyc byte-compiling build/bdist.linux-x86_64/egg/concoct/transform.py to transform.pyc byte-compiling build/bdist.linux-x86_64/egg/concoct/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/concoct/output.py to output.pyc byte-compiling build/bdist.linux-x86_64/egg/concoct/cluster.py to cluster.pyc creating stub loader for vbgmm.so byte-compiling build/bdist.linux-x86_64/egg/vbgmm.py to vbgmm.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts running install_scripts running build_scripts creating build/scripts-2.7 copying and adjusting bin/concoct -> build/scripts-2.7 changing mode of build/scripts-2.7/concoct from 644 to 755 creating build/bdist.linux-x86_64/egg/EGG-INFO/scripts copying build/scripts-2.7/concoct -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/concoct to 755 copying concoct.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying concoct.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt creating dist creating 'dist/concoct-0.4.0-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing concoct-0.4.0-py2.7-linux-x86_64.egg creating /usr/local/lib/python2.7/dist-packages/concoct-0.4.0-py2.7-linux-x86_64.egg Extracting concoct-0.4.0-py2.7-linux-x86_64.egg to /usr/local/lib/python2.7/dist-packages Removing concoct 0.4.1 from easy-install.pth file Adding concoct 0.4.0 to easy-install.pth file Installing concoct script to /usr/local/bin

Installed /usr/local/lib/python2.7/dist-packages/concoct-0.4.0-py2.7-linux-x86_64.egg Processing dependencies for concoct==0.4.0 Searching for sphinx-rtd-theme>=0.1.6 Reading https://pypi.python.org/simple/sphinx-rtd-theme/ Downloading https://pypi.python.org/packages/90/0a/5c162467c8913600f73a2f0f1faef4ef81187f5c829cd2e3930e025d805a/sphinx_rtd_theme-0.2.5b2.tar.gz#md5=5ce05944037d03e1472402bf0931df90 Best match: sphinx-rtd-theme 0.2.5b2 Processing sphinx_rtd_theme-0.2.5b2.tar.gz Writing /tmp/easy_install-oen7UJ/sphinx_rtd_theme-0.2.5b2/setup.cfg Running sphinx_rtd_theme-0.2.5b2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-oen7UJ/sphinx_rtd_theme-0.2.5b2/egg-dist-tmp-sCbmOn creating /usr/local/lib/python2.7/dist-packages/sphinx_rtd_theme-0.2.5b2-py2.7.egg Extracting sphinx_rtd_theme-0.2.5b2-py2.7.egg to /usr/local/lib/python2.7/dist-packages Adding sphinx-rtd-theme 0.2.5b2 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/sphinx_rtd_theme-0.2.5b2-py2.7.egg Searching for Sphinx==1.2.2 Best match: Sphinx 1.2.2 Adding Sphinx 1.2.2 to easy-install.pth file Installing sphinx-apidoc script to /usr/local/bin Installing sphinx-build script to /usr/local/bin Installing sphinx-quickstart script to /usr/local/bin Installing sphinx-autogen script to /usr/local/bin

Using /usr/lib/python2.7/dist-packages Searching for nose==1.3.0 Best match: nose 1.3.0 Processing nose-1.3.0-py2.7.egg nose 1.3.0 is already the active version in easy-install.pth Installing nosetests script to /usr/local/bin Installing nosetests-2.7 script to /usr/local/bin

Using /usr/local/lib/python2.7/dist-packages/nose-1.3.0-py2.7.egg Searching for scikit-learn==0.19.1 Best match: scikit-learn 0.19.1 Processing scikit_learn-0.19.1-py2.7-linux-x86_64.egg scikit-learn 0.19.1 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/scikit_learn-0.19.1-py2.7-linux-x86_64.egg Searching for biopython==1.70 Best match: biopython 1.70 Adding biopython 1.70 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for pandas==0.20.3 Best match: pandas 0.20.3 Adding pandas 0.20.3 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for scipy==0.19.1 Best match: scipy 0.19.1 Adding scipy 0.19.1 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for numpy==1.13.3 Best match: numpy 1.13.3 Adding numpy 1.13.3 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for Cython==0.27.1 Best match: Cython 0.27.1 Adding Cython 0.27.1 to easy-install.pth file Installing cython script to /usr/local/bin Installing cygdb script to /usr/local/bin Installing cythonize script to /usr/local/bin

Using /usr/local/lib/python2.7/dist-packages Searching for Jinja2==2.9.6 Best match: Jinja2 2.9.6 Adding Jinja2 2.9.6 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for docutils==0.11 Best match: docutils 0.11 Adding docutils 0.11 to easy-install.pth file

Using /usr/lib/python2.7/dist-packages Searching for Pygments==2.2.0 Best match: Pygments 2.2.0 Adding Pygments 2.2.0 to easy-install.pth file Installing pygmentize script to /usr/local/bin

Using /usr/local/lib/python2.7/dist-packages Searching for python-dateutil==2.6.1 Best match: python-dateutil _2.6.1 Adding python-dateutil 2.6.1 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for pytz==2012rc0 Best match: pytz 2012rc0 Adding pytz 2012rc0 to easy-install.pth file

Using /usr/lib/python2.7/dist-packages Searching for MarkupSafe==1.0 Best match: MarkupSafe 1.0 Adding MarkupSafe 1.0 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Searching for six==1.11.0 Best match: six 1.11.0 Adding six 1.11.0 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages Finished processing dependencies for concoct==0.4.0

I din't use pip or conda for the download. My gsl version, checked by gsl-config --version, is 1.16.

Any idea? Thanks in advance.

Dani

alneberg commented 6 years ago

Hello,

could you please also specify what source of CONCOCT you're using, a release download (which one?) or have you cloned any branch from github?

Finally, could you please supply the error message given when you try to run it and how you are trying to run it.

Thank you, Johannes

NielQC commented 6 years ago

Of course, here is the information:

$ ./concoct

Traceback (most recent call last):
  File "./concoct", line 6, in <module>
    import vbgmm
ImportError: No module named vbgmm
alneberg commented 6 years ago

Hmm, I'm not sure what's going on here,

could you try calling concoct from your $PATH instead of the actual file?

What happens if you run

which concoct

outside of the bin directory?

NielQC commented 6 years ago

Well, that was the problem, although I don't fully understand why.

$ which concoct
/usr/local/bin/concoct

Calling the conconct from $PATH solved the problem.

$ concoct # running from /home
usage: concoct [-h] [--coverage_file COVERAGE_FILE]
               [--composition_file COMPOSITION_FILE] [-c CLUSTERS]
               [-k KMER_LENGTH] [-l LENGTH_THRESHOLD] [-r READ_LENGTH]
               [--total_percentage_pca TOTAL_PERCENTAGE_PCA] [-b BASENAME]
               [-s SEED] [-i ITERATIONS] [-e EPSILON] [--no_cov_normalization]
               [--no_total_coverage] [-o] [-d] [-v]
concoct: error: No input data supplied, add file(s) using --coverage_file <cov_file> and/or --composition_file <comp_file>

Many thanks for your help and your quickness.

alneberg commented 6 years ago

Great!

I think it's something with how python locates it's internal modules.

I'll close this issue now. Cheers, Johannes

raissafedora commented 5 years ago

Hi, this issue reappeared. Apparently scikit-learn 0.20 removed the vbgmm completely -- so it will turn up the error. May I know the fix ?

alneberg commented 5 years ago

Hi @raissafedora! The vbgmm module referenced here is actually the c-module implemented by concoct and not the one with the same name in scikit-learn. So concoct should not be affected by the removal of it in scikit-learn. If you see this message it's rather likely that something went wrong with the installation of concoct. The current recommendation on how to install concoct can be found here: https://concoct.readthedocs.io/en/latest/installation.html

cwood15 commented 4 years ago

Hello,

I seem to be having the same issue. When I try to run concoct I get the error: Traceback (most recent call last): File "/home/cwood15/anaconda3/envs/concoct_env/bin/concoct", line 6, in import vbgmm ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

My input was: concoct --composition_file contigs_10K.fasta --coverage_file coverage_table.tsv -b concoct_output

I installed concoct using the recommended bioconda installation on https://concoct.readthedocs.io/en/latest/installation.html

Here is my conda environment

packages in environment at /home/cwood15/anaconda3/envs/concoct_env:

#

Name Version Build Channel

_libgcc_mutex 0.1 main biopython 1.78 py38h1e0a361_1 conda-forge blas 1.0 mkl bzip2 1.0.8 h516909a_3 conda-forge c-ares 1.11.0 h470a237_1 bioconda ca-certificates 2020.6.20 hecda079_0 conda-forge certifi 2020.6.20 py38h924ce5b_2 conda-forge concoct 1.1.0 py38h7be5676_2 bioconda cython 0.29.21 py38h348cfbe_1 conda-forge gsl 2.6 hf94e986_0 conda-forge htslib 1.11 hd3b49d5_0 bioconda joblib 0.17.0 py_0 conda-forge krb5 1.17.1 hfafb76e_3 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 libblas 3.9.0 1_h6e990d7_netlib conda-forge libcblas 3.9.0 2_h6e990d7_netlib conda-forge libcurl 7.71.1 hcdd3856_8 conda-forge libdeflate 1.6 h516909a_0 conda-forge libedit 3.1.20191231 h14c3975_1
libev 4.33 h516909a_1 conda-forge libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.5.0 hae1eefd_17 conda-forge libgfortran4 7.5.0 hae1eefd_17 conda-forge liblapack 3.9.0 2_h6e990d7_netlib conda-forge libnghttp2 1.41.0 hab1572f_1 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 llvm-openmp 8.0.1 hc9558a2_0 conda-forge ncurses 6.2 he6710b0_1 nose 1.3.7 py_1006 conda-forge numpy 1.19.4 py38hf0fd68c_0 conda-forge openmp 8.0.1 0 conda-forge openssl 1.1.1h h516909a_0 conda-forge pandas 1.1.4 py38h0ef3d22_0 conda-forge pip 20.2.4 py_0 conda-forge python 3.8.5 h7579374_1 python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2020.4 pyhd8ed1ab_0 conda-forge readline 8.0 h7b6447c_0 samtools 1.11 h6270b1f_0 bioconda scikit-learn 0.23.2 py38h5d63f67_2 conda-forge scipy 1.5.3 py38h828c644_0 conda-forge setuptools 49.6.0 py38h924ce5b_2 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.33.0 h62c20be_0 threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge tk 8.6.10 hbc83047_0 tzdata 2020d h14c3975_0 wheel 0.35.1 py_0 xz 5.2.5 h7b6447c_0 zlib 1.2.11 h7b6447c_3

Any help would be greatly appreciated! Thank you!

Claudia

jtamames commented 3 years ago

Hello Claudia

try conda install mkl in your concoct conda env.

Best, Javier

valzip commented 3 years ago

Hello, I can confirm that at vanilla installation of concoct (v1.1.0 py38h7be5676_2) through conda (conda v4.9.2, python v3.8.3.final.0) there is an error in dependencies - missing vbgmm.

Traceback (most recent call last):
  File "/home/$USER/miniconda3/envs/concoct_env/bin/concoct", line 6, in <module>
    import vbgmm

Solution of @jtamames works very well. Even when called from bash scripts. 

conda activate concoct_env
conda install mkl
conda deactivate
dportik commented 1 year ago

Also bumped in to this recently with a conda environment install for concoct as part of a snakemake workflow.

The following yaml recipe can be used to create the correct environment and avoid this issue (as well as another by using python 3.8):

name: concoct_env
channels:
- bioconda
- conda-forge
- defaults
dependencies:
- concoct == 1.1.0
- python == 3.8
- mkl

If you are using this outside of snakemake, you can create the environment with: 

conda env create -f concoct_env.yaml
conda activate concoct_env

Assuming you've named the yaml file as concoct_env.yaml.