search

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
31.5k stars 3.65k forks source link

Weird behaviour under macOS #2868

Open Naozumi520 opened 1 year ago

Naozumi520 commented 1 year ago

Summary

Cannot install under macOS, throw error during Installation.

Platform

OS: macOS 13.3.1

Faiss version: 1.7.0

Installed from: pip

Faiss compilation options: none

Reproduction instructions

run pip3 install faiss-cpu==1.7.0

Throw error:

naozumi@Naozumis-MacBook-Pro RVC-beta % pip3 install faiss-cpu==1.7.0  
Collecting faiss-cpu==1.7.0
  Using cached faiss-cpu-1.7.0.tar.gz (34 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: faiss-cpu
  Building wheel for faiss-cpu (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'faiss._swigfaiss' extension
      swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
      swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
      faiss/faiss/python/swigfaiss.i:241: Error: Unable to find 'faiss/impl/platform_macros.h'
      faiss/faiss/python/swigfaiss.i:245: Error: Unable to find 'faiss/utils/ordered_key_value.h'
      faiss/faiss/python/swigfaiss.i:246: Error: Unable to find 'faiss/utils/Heap.h'
      faiss/faiss/python/swigfaiss.i:251: Error: Unable to find 'faiss/utils/AlignedTable.h'
      faiss/faiss/python/swigfaiss.i:252: Error: Unable to find 'faiss/utils/partitioning.h'
      faiss/faiss/python/swigfaiss.i:253: Error: Unable to find 'faiss/utils/hamming.h'
      faiss/faiss/python/swigfaiss.i:356: Error: Unable to find 'faiss/utils/utils.h'
      faiss/faiss/python/swigfaiss.i:357: Error: Unable to find 'faiss/utils/distances.h'
      faiss/faiss/python/swigfaiss.i:358: Error: Unable to find 'faiss/utils/random.h'
      faiss/faiss/python/swigfaiss.i:360: Error: Unable to find 'faiss/MetricType.h'
      faiss/faiss/python/swigfaiss.i:363: Error: Unable to find 'faiss/Index.h'
      faiss/faiss/python/swigfaiss.i:364: Error: Unable to find 'faiss/Clustering.h'
      faiss/faiss/python/swigfaiss.i:366: Error: Unable to find 'faiss/utils/extra_distances.h'
      faiss/faiss/python/swigfaiss.i:370: Error: Unable to find 'faiss/impl/ProductQuantizer.h'
      faiss/faiss/python/swigfaiss.i:372: Error: Unable to find 'faiss/VectorTransform.h'
      faiss/faiss/python/swigfaiss.i:373: Error: Unable to find 'faiss/IndexPreTransform.h'
      faiss/faiss/python/swigfaiss.i:374: Error: Unable to find 'faiss/IndexFlat.h'
      faiss/faiss/python/swigfaiss.i:375: Error: Unable to find 'faiss/IndexRefine.h'
      faiss/faiss/python/swigfaiss.i:376: Error: Unable to find 'faiss/IndexLSH.h'
      faiss/faiss/python/swigfaiss.i:377: Error: Unable to find 'faiss/impl/PolysemousTraining.h'
      faiss/faiss/python/swigfaiss.i:378: Error: Unable to find 'faiss/IndexPQ.h'
      faiss/faiss/python/swigfaiss.i:379: Error: Unable to find 'faiss/impl/io.h'
      faiss/faiss/python/swigfaiss.i:381: Error: Unable to find 'faiss/invlists/InvertedLists.h'
      faiss/faiss/python/swigfaiss.i:382: Error: Unable to find 'faiss/invlists/InvertedListsIOHook.h'
      faiss/faiss/python/swigfaiss.i:384: Error: Unable to find 'faiss/invlists/BlockInvertedLists.h'
      faiss/faiss/python/swigfaiss.i:385: Error: Unable to find 'faiss/invlists/DirectMap.h'
      faiss/faiss/python/swigfaiss.i:388: Error: Unable to find 'faiss/IndexIVF.h'
      faiss/faiss/python/swigfaiss.i:393: Error: Unable to find 'faiss/IVFlib.h'
      faiss/faiss/python/swigfaiss.i:394: Error: Unable to find 'faiss/impl/ScalarQuantizer.h'
      faiss/faiss/python/swigfaiss.i:395: Error: Unable to find 'faiss/IndexScalarQuantizer.h'
      faiss/faiss/python/swigfaiss.i:396: Error: Unable to find 'faiss/IndexIVFSpectralHash.h'
      faiss/faiss/python/swigfaiss.i:397: Error: Unable to find 'faiss/impl/HNSW.h'
      faiss/faiss/python/swigfaiss.i:398: Error: Unable to find 'faiss/IndexHNSW.h'
      faiss/faiss/python/swigfaiss.i:399: Error: Unable to find 'faiss/IndexIVFFlat.h'
      faiss/faiss/python/swigfaiss.i:404: Error: Unable to find 'faiss/invlists/OnDiskInvertedLists.h'
      faiss/faiss/python/swigfaiss.i:407: Error: Unable to find 'faiss/impl/lattice_Zn.h'
      faiss/faiss/python/swigfaiss.i:408: Error: Unable to find 'faiss/IndexLattice.h'
      faiss/faiss/python/swigfaiss.i:411: Error: Unable to find 'faiss/IndexIVFPQ.h'
      faiss/faiss/python/swigfaiss.i:412: Error: Unable to find 'faiss/IndexIVFPQR.h'
      faiss/faiss/python/swigfaiss.i:413: Error: Unable to find 'faiss/Index2Layer.h'
      faiss/faiss/python/swigfaiss.i:415: Error: Unable to find 'faiss/IndexPQFastScan.h'
      faiss/faiss/python/swigfaiss.i:416: Error: Unable to find 'faiss/IndexIVFPQFastScan.h'
      faiss/faiss/python/swigfaiss.i:417: Error: Unable to find 'faiss/utils/quantize_lut.h'
      faiss/faiss/python/swigfaiss.i:419: Error: Unable to find 'faiss/IndexBinary.h'
      faiss/faiss/python/swigfaiss.i:420: Error: Unable to find 'faiss/IndexBinaryFlat.h'
      faiss/faiss/python/swigfaiss.i:421: Error: Unable to find 'faiss/IndexBinaryIVF.h'
      faiss/faiss/python/swigfaiss.i:422: Error: Unable to find 'faiss/IndexBinaryFromFloat.h'
      faiss/faiss/python/swigfaiss.i:423: Error: Unable to find 'faiss/IndexBinaryHNSW.h'
      faiss/faiss/python/swigfaiss.i:424: Error: Unable to find 'faiss/IndexBinaryHash.h'
      faiss/faiss/python/swigfaiss.i:426: Error: Unable to find 'faiss/impl/ThreadedIndex.h'
      faiss/faiss/python/swigfaiss.i:430: Error: Unable to find 'faiss/IndexShards.h'
      faiss/faiss/python/swigfaiss.i:434: Error: Unable to find 'faiss/IndexReplicas.h'
      faiss/faiss/python/swigfaiss.i:438: Error: Unable to find 'faiss/MetaIndexes.h'
      faiss/faiss/python/swigfaiss.i:453: Error: Unable to find 'faiss/impl/AuxIndexStructures.h'
      faiss/faiss/python/swigfaiss.i:639: Error: Unable to find 'faiss/index_io.h'
      faiss/faiss/python/swigfaiss.i:640: Error: Unable to find 'faiss/clone_index.h'
      faiss/faiss/python/swigfaiss.i:644: Error: Unable to find 'faiss/AutoTune.h'
      faiss/faiss/python/swigfaiss.i:645: Error: Unable to find 'faiss/index_factory.h'
      faiss/faiss/python/swigfaiss.i:646: Error: Unable to find 'faiss/MatrixStats.h'
      error: command '/usr/local/bin/swig' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for faiss-cpu
  Running setup.py clean for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects
naozumi@Naozumis-MacBook-Pro RVC-beta %
Naozumi520 commented 1 year ago

After using conda as the installer, I've succeeded to install faiss. But it made weird errors and the same using conda-forge

(svc01) naozumi@Naozumis-MBP RVC-beta % python infer-web.py                                  
Traceback (most recent call last):
  File "infer-web.py", line 7, in <module>
    import faiss
  File "/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/loader.py", line 65, in <module>
    from .swigfaiss import *
  File "/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/swigfaiss.py", line 13, in <module>
    from . import _swigfaiss
ImportError: dlopen(/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/_swigfaiss.so, 0x0002): Library not loaded: @rpath/libmkl_intel_lp64.1.dylib
  Referenced from: <4A025BD7-6F1F-3CD5-B319-DE475BB38238> /usr/local/anaconda3/envs/svc01/lib/libfaiss.dylib
  Reason: tried: '/usr/local/anaconda3/envs/svc01/lib/libmkl_intel_lp64.1.dylib' (no such file), '/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/../../../libmkl_intel_lp64.1.dylib' (no such file), '/usr/local/anaconda3/envs/svc01/lib/python3.8/site-packages/faiss/../../../libmkl_intel_lp64.1.dylib' (no such file), '/usr/local/anaconda3/envs/svc01/bin/../lib/libmkl_intel_lp64.1.dylib' (no such file), '/usr/local/anaconda3/envs/svc01/bin/../lib/libmkl_intel_lp64.1.dylib' (no such file), '/usr/local/lib/libmkl_intel_lp64.1.dylib' (no such file), '/usr/lib/libmkl_intel_lp64.1.dylib' (no such file, not in dyld cache)
mdouze commented 1 year ago

There is currently an issue with MKL on the Mac. @algoriddle what is the current status?

JingweiZuo commented 1 year ago

The same issue happened to me when running on the Mac. I have a M1 chip.

Do you have any updates for MKL on the Mac? @mdouze @algoriddle

Thank you!

liyuyuan6969 commented 1 year ago

+1

ghost commented 1 year ago

+1

chongzluong commented 1 year ago

+1 - got around this by just conda install -c conda-forge faiss-cpu==1.7.3 in the interim

luchko commented 1 year ago

+1

CharlesDDDD commented 1 year ago

+1

dalssoft commented 10 months ago

the same! +1

> sw_vers

ProductName: macOS ProductVersion: 14.2.1 BuildVersion: 23C71

> python --version

Python 3.12.0

> pip --version

pip 23.3.1 from /Users/xxx/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pip (python 3.12)

InSuperposition commented 10 months ago

+1

> pip wheel --no-cache-dir --use-pep517 "faiss-cpu (==1.7.4)"
> sw_vers

ProductName:            macOS
ProductVersion:         14.2.1
BuildVersion:           23C71

> python --version

Python 3.12.0

> pip --version

pip 23.3.1 from /Users/xxx/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pip (python 3.12)
InSuperposition commented 10 months ago

There is an open PR https://github.com/facebookresearch/faiss/pull/3191 for an issue using faiss with python 3.12

dalssoft commented 9 months ago

It worked after changing python to v3.10 and installing FAISS using brew

> brew install faiss --build-from-source