faiss searches for wrong library name

[Randl]

Summary

faiss searches for the following libraries

libmkl_core.so.1
libmkl_gnu_thread.so.1
libmkl_intel_lp64.so.1

but in fact the ones installed by conda have names ending in .2. It leads to crashes with errors like

Traceback (most recent call last):
  File "/home/evgeniizh/research/xtreme/third_party/evaluate_retrieval.py", line 28, in <module>
    import faiss
  File "/opt/conda/envs/tmp3/lib/python3.10/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/opt/conda/envs/tmp3/lib/python3.10/site-packages/faiss/loader.py", line 65, in <module>
    from .swigfaiss import *
  File "/opt/conda/envs/tmp3/lib/python3.10/site-packages/faiss/swigfaiss.py", line 13, in <module>
    from . import _swigfaiss
ImportError: libmkl_core.so.1: cannot open shared object file: No such file or directory

symlinking them like

ln -s /opt/conda/envs/tmp3/lib/libmkl_core.so.2 /opt/conda/envs/tmp3/lib/libmkl_core.so.1

for all three fixes the problem

Platform

OS: Ubuntu 20.04.5 LTS

Faiss version: faiss-gpu 1.7.4 (py3.10_hc0239a3_0_cuda11.4, pytorch)

Installed from: pytorch channel of conda

Faiss compilation options:

Running on:

• [ ] CPU
• [ ] GPU

Interface:

• [ ] C++
• [x] Python

Reproduction instructions

Run conda env create -n tmp -f conda-req.txt with the following conda-req.txt:

pytorch==2.0*=*cu*
transformers
faiss-gpu>=1.7
pandas>=1.5
python=3.10
scipy>=1.10
svn
sentencepiece
seqeval>=1.2
blas=*=*mkl*

In python, try, for example, from faiss import _swigfaiss_gpu.

[algoriddle]

I changed the conda package to require mkl =2023 going forward both at build and runtime. Previously it was built with 2021, but defined mkl >= 2021 as its runtime requirement. I'm crossing fingers this fixes the problem. You can try this with the nightly build conda install faiss-gpu -c pytorch/label/nightly.

[Randl]

with faiss-gpu version pytorch/label/nightly/linux-64::faiss-gpu-1.7.4-py3.10_hc0239a3_12_cuda11.4_nightly I get the same error

[binodgyawali]

I had the same issue with faiss-cpu=1.7.4. I was able to import faiss after symlinking the three libraries.

libmkl_core.so.1
libmkl_gnu_thread.so.1
libmkl_intel_lp64.so.1