Instalation issue

[luhugerth]

I think this is related to this issue: https://sourceforge.net/p/concoct/mailman/message/32866851/

I didn't get any errors when creating the env, installing the libraries or running setup.py. However:

$ bin/concoct -h Traceback (most recent call last): File "/home/luisah/miniconda/envs/concoct-env/bin/concoct", line 4, in import('pkg_resources').run_script('concoct==0.4.1', 'concoct') File "/home/luisah/miniconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 744, in run_script File "/home/luisah/miniconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 1499, in run_script File "/pica/h1/luisah/miniconda/envs/concoct-env/lib/python2.7/site-packages/concoct-0.4.1-py2.7-linux-x86_64.egg/EGG-INFO/scripts/concoct", line 8, in from concoct.output import Output File "/home/luisah/miniconda/envs/concoct-env/lib/python2.7/site-packages/concoct-0.4.1-py2.7-linux-x86_64.egg/concoct/output.py", line 11, in import pandas as p File "/home/luisah/miniconda/lib/python2.7/site-packages/pandas-0.19.2-py2.7-linux-x86_64.egg/pandas/init.py", line 31, in "the C extensions first.".format(module)) ImportError: C extension: /home/luisah/miniconda/lib/python2.7/site-packages/pandas-0.19.2-py2.7-linux-x86_64.egg/pandas/hashtable.so: undefined symbol: PyUnicodeUCS2_Format not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

So I do what the message tells me:

$ python setup.py build_ext --inplace --force running build_ext building 'vbgmm' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ic-concoct -I/home/luisah/miniconda/envs/concoct-env/include/python2.7 -c c-concoct/vbgmmmodule.c -o build/temp.linux-x86_64-2.7/c-concoct/vbgmmmodule.o In file included from /home/luisah/miniconda/envs/concoct-env/include/python2.7/Python.h:8:0, from c-concoct/vbgmmmodule.c:29: /home/luisah/miniconda/envs/concoct-env/include/python2.7/pyconfig.h:1179:0: warning: "_POSIX_C_SOURCE" redefined [enabled by default]

define _POSIX_C_SOURCE 200112L

^ In file included from /usr/include/stdlib.h:24:0, from c-concoct/vbgmmmodule.c:2: /usr/include/features.h:231:0: note: this is the location of the previous definition

define _POSIX_C_SOURCE 200809L

^ c-concoct/vbgmmmodule.c: In function ‘driver’: c-concoct/vbgmmmodule.c:121:28: warning: assignment discards ‘const’ qualifier from pointer target type [enabled by default] ptBestCluster->szCOutFile = szFileStub; ^ c -concoct/vbgmmmodule.c:147:5: warning: implicit declaration of function ‘gsl_blas_dgemm’ [-Wimplicit-function-declaration] gsl_blas_dgemm (CblasNoTrans,CblasNoTrans,1.0,tData.ptTMatrix,ptBestCluster->aptSigma[k],0.0,ptTemp); ^ c-concoct/vbgmmmodule.c: In function ‘performMStep’: c-concoct/vbgmmmodule.c:669:17: warning: unused variable ‘ptSigmaMatrix’ [-Wunused-variable] gsl_matrix* ptSigmaMatrix = gsl_matrix_alloc(nD,nD); ^ c-concoct/vbgmmmodule.c: In function ‘calcVBL’: c-concoct/vbgmmmodule.c:1401:7: warning: implicit declaration of function ‘gsl_blas_dsymv’ [-Wimplicit-function-declaration] gsl_blas_dsymv (CblasLower, 1.0, ptCluster->aptSigma[k], ptDiff, 0.0, ptR); ^ c-concoct/vbgmmmodule.c:1403:7: warning: implicit declaration of function ‘gsl_blas_ddot’ [-Wimplicit-function-declaration] gsl_blas_ddot (ptDiff, ptR, &dT2); ^ c-concoct/vbgmmmodule.c: In function ‘writeTMeans’: c-concoct/vbgmmmodule.c:1644:7: warning: implicit declaration of function ‘gsl_blas_dgemv’ [-Wimplicit-function-declaration] gsl_blas_dgemv (CblasNoTrans, 1.0,ptData->ptTMatrix,ptVector,0.0, ptTVector); ^ c-concoct/vbgmmmodule.c: In function ‘runRThreads’: c-concoct/vbgmmmodule.c:1691:15: warning: variable ‘iret’ set but not used [-Wunused-but-set-variable] int iret[N_RTHREADS]; ^ gcc -pthread -shared build/temp.linux-x86_64-2.7/c-concoct/vbgmmmodule.o -L/home/luisah/miniconda/envs/concoct-env/lib -lgsl -lgslcblas -lpython2.7 -o build/lib.linux-x86_64-2.7/vbgmm.so copying build/lib.linux-x86_64-2.7/vbgmm.so ->

But still get the same error as before:

$ bin/concoct Traceback (most recent call last): File "bin/concoct", line 8, in from concoct.output import Output File "/home/luisah/miniconda/envs/concoct-env/lib/python2.7/site-packages/concoct-0.4.1-py2.7-linux-x86_64.egg/concoct/output.py", line 11, in import pandas as p File "/home/luisah/miniconda/lib/python2.7/site-packages/pandas-0.19.2-py2.7-linux-x86_64.egg/pandas/init.py", line 31, in "the C extensions first.".format(module)) ImportError: C extension: /home/luisah/miniconda/lib/python2.7/site-packages/pandas-0.19.2-py2.7-linux-x86_64.egg/pandas/hashtable.so: undefined symbol: PyUnicodeUCS2_Format not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

Any ideas??

[alneberg]

Hello,

I think you're mixing up the concoct installation with the pandas installation, the first error message "setup.py build_ext --inplace --force' to build the C extensions first." is output from pandas so installing concoct with those parameters would not make a difference. The problem seems to be with the pandas installation. Could you try to reinstall pandas using conda?

[luhugerth]

Tried it, same thing. Tried creating a new env from scratch and repeating the operation, same thing. Could it depend on the version of either pandas or one of its dependencies? Are there versions you recommend?

No need to answer during the holidays ;)

[alneberg]

Could it be related to this:

https://stackoverflow.com/a/39342730

Do you get the same error by just running

import pandas

in a python session? That would verify that this is a pandas installation issue.

[luhugerth]

Hi, You're right, it was the Pandas installation, which in turn was related to the fact that my Miniconda was too old, combined with messy $PATH and $PYTHONPATH variables.

When that was fixed I went back to getting the good old 'no module named vbgmm' error, but that was due to our new server not having gsl activated per default. Activating gsl within the virtualenv and then rerunning the install script finally fixed it.

Thanks for the support, you can close the issue now :)

[alneberg]

Yey!

Thanks for the detailed explanation of the solution as well, something that others might benefit from.