Python interface compilation issues

[alexjwilliams]

Hello,

I'm having some trouble compiling the python interface. I don't have much experience with make and compiling C++ code, so I apologize if what I ask about has an obvious fix.

I was able to compile the CPU version of faiss and the tests ran without errors.

To install the python interface I used the following line in makefile.inc: PYTHONCFLAGS=-I/home/paperspace/anaconda3/envs/py34/include/python3.4m/ -I/home/paperspace/anaconda3/envs/py34/lib/python3.4/site-packages/numpy/core/include/. Note that I am using python in a conda environment.

When I run make py, I get the following output: AuxIndexStructures.h:118: Warning 325: Nested struct not currently supported (Buffer ignored) AuxIndexStructures.h:165: Warning 325: Nested struct not currently supported (QueryResult ignored) g++ -I. -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -g -fPIC -fopenmp -I/home/paperspace/anaconda3/envs/py34/include/python3.4m/ -I/home/paperspace/anaconda3/envs/py34/lib/python3.4/site-packages/numpy/core/include/ -shared \ -o python/_swigfaiss.so python/swigfaiss_wrap.cxx libfaiss.a /usr/lib/libopenblas.so.0 /usr/lib/lapack/liblapack.so.3.0 cp python/_swigfaiss.so python/swigfaiss.py . I included this just in case those two warnings are of issue.

When I run python -c "import faiss" to test the installation when I am in ~/git/faiss, I get the following output:

(py34) paperspace@xxxxx:~/git/faiss$ python -c "import faiss" Failed to load GPU Faiss: No module named 'swigfaiss_gpu' Faiss falling back to CPU-only. Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/paperspace/git/faiss/faiss.py", line 179, in <module> if inspect.isclass(obj): File "/home/paperspace/anaconda3/envs/py34/lib/python3.4/inspect.py", line 83, in isclass return isinstance(object, type) NameError: Unknown C global variable

If I navigate to a directory other than ~/git/faiss and then execute python -c "import faiss", I get the following: Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named 'faiss'. EDIT: This means that faiss is not in the python path. Is there any way for make to automatically add faiss.py to the python path during the installation process? This could save future users some work.

I would be grateful if someone could help me address these two errors.

My complete makefile.inc is here: https://gist.github.com/alexwms/f8d9c254326d4d0a7d1d90b8ee96bafa.

I am on Ubuntu 14.04 and I have g++ version 4.8.5 installed.

Thanks, Alex

[alexjwilliams]

I ran ldd -r _swigfaiss.so to get a list of all the undefined symbols. The output is below. I do not know what to do to fix this.

EDIT: According to #119 this is normal.

paperspace@xxxxxxx:~/git/faiss$ ldd -r _swigfaiss.so linux-vdso.so.1 => (0x00007ffe921e5000) libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f99e3d7a000) liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f99e35de000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f99e32cb000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f99e2fc5000) libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f99e2d97000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f99e2b7f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f99e2961000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99e2599000) libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f99e0d8f000) libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f99e0a68000) /lib64/ld-linux-x86-64.so.2 (0x000055e960474000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f99e0828000) undefined symbol: PyExc_ValueError (./_swigfaiss.so) undefined symbol: PyExc_IndexError (./_swigfaiss.so) undefined symbol: PyExc_SyntaxError (./_swigfaiss.so) undefined symbol: PyCapsule_Type (./_swigfaiss.so) undefined symbol: _Py_NoneStruct (./_swigfaiss.so) undefined symbol: PyExc_AttributeError (./_swigfaiss.so) undefined symbol: PyExc_IOError (./_swigfaiss.so) undefined symbol: PyExc_MemoryError (./_swigfaiss.so) undefined symbol: PyType_Type (./_swigfaiss.so) undefined symbol: PyExc_OverflowError (./_swigfaiss.so) undefined symbol: PyExc_NameError (./_swigfaiss.so) undefined symbol: PyExc_ZeroDivisionError (./_swigfaiss.so) undefined symbol: PyExc_NotImplementedError (./_swigfaiss.so) undefined symbol: PyExc_RuntimeError (./_swigfaiss.so) undefined symbol: _Py_NotImplementedStruct (./_swigfaiss.so) undefined symbol: PyBaseObject_Type (./_swigfaiss.so) undefined symbol: PyExc_TypeError (./_swigfaiss.so) undefined symbol: PyObject_GenericGetAttr (./_swigfaiss.so) undefined symbol: PyFloat_Type (./_swigfaiss.so) undefined symbol: PyExc_SystemError (./_swigfaiss.so) undefined symbol: PyExc_ImportError (./_swigfaiss.so) undefined symbol: PyTuple_SetItem (./_swigfaiss.so) undefined symbol: PyLong_AsLong (./_swigfaiss.so) undefined symbol: PyObject_GetAttr (./_swigfaiss.so) undefined symbol: PyImport_AddModule (./_swigfaiss.so) undefined symbol: PyObject_Call (./_swigfaiss.so) undefined symbol: PyObject_Init (./_swigfaiss.so) undefined symbol: PyInstanceMethod_New (./_swigfaiss.so) undefined symbol: PyObject_GetAttrString (./_swigfaiss.so) undefined symbol: PyType_IsSubtype (./_swigfaiss.so) undefined symbol: PyModule_GetDict (./_swigfaiss.so) undefined symbol: PyModule_AddObject (./_swigfaiss.so) undefined symbol: PyUnicode_InternFromString (./_swigfaiss.so) undefined symbol: PyTuple_New (./_swigfaiss.so) undefined symbol: PyObject_SetAttr (./_swigfaiss.so) undefined symbol: PyObject_IsInstance (./_swigfaiss.so) undefined symbol: PyEval_RestoreThread (./_swigfaiss.so) undefined symbol: PyDict_SetItemString (./_swigfaiss.so) undefined symbol: PyUnicode_FromFormat (./_swigfaiss.so) undefined symbol: _PyObject_New (./_swigfaiss.so) undefined symbol: PyErr_Format (./_swigfaiss.so) undefined symbol: PyFloat_FromDouble (./_swigfaiss.so) undefined symbol: PyFloat_AsDouble (./_swigfaiss.so) undefined symbol: PyImport_ImportModule (./_swigfaiss.so) undefined symbol: PyUnicode_FromString (./_swigfaiss.so) undefined symbol: PyErr_Clear (./_swigfaiss.so) undefined symbol: Py_DecRef (./_swigfaiss.so) undefined symbol: PyCapsule_Import (./_swigfaiss.so) undefined symbol: PyLong_FromVoidPtr (./_swigfaiss.so) undefined symbol: PyErr_SetString (./_swigfaiss.so) undefined symbol: PyCapsule_New (./_swigfaiss.so) undefined symbol: PyArg_ParseTuple (./_swigfaiss.so) undefined symbol: PyCapsule_GetPointer (./_swigfaiss.so) undefined symbol: PyObject_Free (./_swigfaiss.so) undefined symbol: PyType_Ready (./_swigfaiss.so) undefined symbol: PyObject_Size (./_swigfaiss.so) undefined symbol: PyEval_SaveThread (./_swigfaiss.so) undefined symbol: PyLong_FromLong (./_swigfaiss.so) undefined symbol: PyObject_IsTrue (./_swigfaiss.so) undefined symbol: PyModule_Create2 (./_swigfaiss.so) undefined symbol: PyLong_AsUnsignedLong (./_swigfaiss.so) undefined symbol: PyErr_Occurred (./_swigfaiss.so) undefined symbol: PyObject_CallFunctionObjArgs (./_swigfaiss.so) undefined symbol: PyLong_FromUnsignedLong (./_swigfaiss.so) undefined symbol: PyArg_UnpackTuple (./_swigfaiss.so) undefined symbol: PyObject_Malloc (./_swigfaiss.so) undefined symbol: PyErr_Print (./_swigfaiss.so) undefined symbol: PyUnicode_Concat (./_swigfaiss.so) undefined symbol: PyUnicode_AsUTF8String (./_swigfaiss.so) undefined symbol: PyUnicode_FromStringAndSize (./_swigfaiss.so) undefined symbol: PyBytes_AsStringAndSize (./_swigfaiss.so) undefined symbol: PyBool_FromLong (./_swigfaiss.so)

[alexjwilliams]

I was able to get the Python front end working. I was originally using swig 2.0.11. Updating to swig 3.0 solved my problems.

[mdouze]

Thanks, closing issue.

[toretak]

I have same problem, but compiled with (g++) SWIG 3.0.8-0ubuntu3 (AWS EC2 machine).. any ideas please?

Thanks

[mdouze]

Hi Please give more context.

[toretak]

Hi, I'v successfuly compiled faiss to library (cpu and gpu), there were few warnings, but it seems to be working

make

ubuntu@ip-172-31-46-162:~/faiss$ make -j 10
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c hamming.cpp -o hamming.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c utils.cpp -o utils.o  -DFINTEGER=int
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexFlat.cpp -o IndexFlat.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexIVF.cpp -o IndexIVF.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexLSH.cpp -o IndexLSH.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexPQ.cpp -o IndexPQ.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexIVFPQ.cpp -o IndexIVFPQ.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c Clustering.cpp -o Clustering.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c Heap.cpp -o Heap.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c VectorTransform.cpp -o VectorTransform.o  -DFINTEGER=int
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c index_io.cpp -o index_io.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c PolysemousTraining.cpp -o PolysemousTraining.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c MetaIndexes.cpp -o MetaIndexes.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c Index.cpp -o Index.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c ProductQuantizer.cpp -o ProductQuantizer.o  -DFINTEGER=int
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c AutoTune.cpp -o AutoTune.o
AutoTune.cpp: In member function ‘virtual void faiss::ParameterSpace::initialize(const faiss::Index*)’:
AutoTune.cpp:309:22: warning: unused variable ‘ix’ [-Wunused-variable]
     const classname *ix = dynamic_cast<const classname *>(index)
                      ^
AutoTune.cpp:389:9: note: in expansion of macro ‘DC’
     if (DC (IndexIVFPQR)) {
         ^
AutoTune.cpp: In function ‘faiss::Index* faiss::index_factory(int, const char*, faiss::MetricType)’:
AutoTune.cpp:706:40: warning: unused variable ‘pq_m’ [-Wunused-variable]
         int d_out, opq_M, nbit, M, M2, pq_m, ncent;
                                        ^
AutoTune.cpp:706:46: warning: unused variable ‘ncent’ [-Wunused-variable]
         int d_out, opq_M, nbit, M, M2, pq_m, ncent;
                                              ^
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c AuxIndexStructures.cpp -o AuxIndexStructures.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexScalarQuantizer.cpp -o IndexScalarQuantizer.o
MetaIndexes.cpp:286:6: warning: ‘void faiss::{anonymous}::translate_labels(long int, faiss::{anonymous}::idx_t*, long int)’ defined but not used [-Wunused-function]
 void translate_labels (long n, idx_t *labels, long translation)
      ^
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c FaissException.cpp -o FaissException.o
g++ -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -c IndexHNSW.cpp -o IndexHNSW.o  -DFINTEGER=int
IndexHNSW.cpp: In member function ‘void faiss::ReconstructFromNeighbors::estimate_code(const float*, faiss::ReconstructFromNeighbors::storage_idx_t, uint8_t*) const’:
IndexHNSW.cpp:1545:13: warning: unused variable ‘d1’ [-Wunused-variable]
         int d1 = d0 + dsub;
             ^
IndexHNSW.cpp: In function ‘int faiss::{anonymous}::search_from_candidates_2(const faiss::HNSW&, faiss::{anonymous}::DistanceComputer&, int, faiss::{anonymous}::idx_t*, float*, faiss::{anonymous}::MinimaxHeap&, faiss::VisitedTable&, int, int)’:
IndexHNSW.cpp:2001:15: warning: unused variable ‘d’ [-Wunused-variable]
         float d = candidates.dis[i];
               ^
ar r libfaiss.a hamming.o utils.o IndexFlat.o IndexIVF.o IndexLSH.o IndexPQ.o IndexIVFPQ.o Clustering.o Heap.o VectorTransform.o index_io.o PolysemousTraining.o MetaIndexes.o Index.o ProductQuantizer.o AutoTune.o AuxIndexStructures.o IndexScalarQuantizer.o FaissException.o IndexHNSW.o
ar: creating libfaiss.a
g++ -o tests/demo_ivfpq_indexing -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp tests/demo_ivfpq_indexing.cpp libfaiss.a -g -fPIC  -fopenmp /usr/lib/libopenblas.so.0

examples in /test directory works

make py

ubuntu@ip-172-31-46-162:~/faiss$ make py
swig -python -c++ -Doverride= -o python/swigfaiss_wrap.cxx swigfaiss.swig
g++ -I. -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -g -fPIC  -fopenmp -I/usr/include/python3.5/ -I/usr/local/lib/python3.6/dist-packages/numpy/core/include/ -shared \
-o python/_swigfaiss.so python/swigfaiss_wrap.cxx libfaiss.a /usr/lib/libopenblas.so.0
cp python/_swigfaiss.so python/swigfaiss.py .

(looks good), but python -c "import faiss"

Failed to load GPU Faiss: ./_swigfaiss_gpu.so: undefined symbol: PyInstanceMethod_New
Faiss falling back to CPU-only.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "faiss.py", line 28, in <module>
    from swigfaiss import *
  File "swigfaiss.py", line 28, in <module>
    _swigfaiss = swig_import_helper()
  File "swigfaiss.py", line 24, in swig_import_helper
    _mod = imp.load_module('_swigfaiss', fp, pathname, description)
ImportError: ./_swigfaiss.so: undefined symbol: PyInstanceMethod_New

in makefine.inc a have these flags

PYTHONCFLAGS=-I/usr/include/python3.5/ -I/usr/local/lib/python3.5/dist-packages/numpy/core/include/

what else could help?

Thank You very much!

[arkogupta]

@alexwms @mdouze I am getting the exact same error. No matter which directory I am in.

(dev) arkogupta@Ag16:/usr/local/faiss$ python -c "import faiss"
Failed to load GPU Faiss: No module named 'swigfaiss_gpu'
Faiss falling back to CPU-only.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/faiss/faiss.py", line 248, in <module>
    if inspect.isclass(obj):
  File "/home/arkogupta/anaconda3/envs/dev/lib/python3.6/inspect.py", line 79, in isclass
    return isinstance(object, type)
NameError: Unknown C global variable

Also I have the latest version of swig installed(3.0.12) !

[mdouze]

Hi So what error are you getting? the inspect.isclass or the missing PyInstanceMethod_New?

[arkogupta]

Hi ,

I am getting the inspect.isclass(obj) error. The error message can be seen in the previous comment. I am running Ubuntu 14.04 and using python 3.6 in an anaconda environment.

Thanks!

[tuglat]

I'm having the same problem, i.e., the inspect.isclass(obj) error. Same error message as previous posters. Running RedHat 7.4, using python 3.6 in an anaconda environment. Using swig 3.0.8.

Any hints?

Thanks

[MichalLeonBorsuk]

Hi,

Look here for a similar problem: https://www.riverbankcomputing.com/pipermail/pyqt/2014-July/034561.html

[mdouze]

No activity, closing.