faiss 1.8.0.post1 crashes on Mac M2

[adampauls]

Summary

Something as simple as this:

import faiss

faiss.omp_set_num_threads(4)

gives

Fatal Python error: Aborted

Thread 0x000000016eb4f000 (most recent call first):
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331 
  ...

Thread 0x000000016db43000 (most recent call first):
  ...

Thread 0x000000016cb37000 (most recent call first):
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331 
  ...

Current thread 0x000000020534cf40 (most recent call first):
  File "/Users/adpauls/sc/git/precog/.venv/lib/python3.11/site-packages/faiss/swigfaiss.py", line 11441 in omp_set_num_threads
  ...

So does

def test_just_faiss():
    vector_length = 32
    index = faiss.IndexFlatL2(vector_length)
    index.add(np.random.rand(1, vector_length))
    queries = np.random.rand(1, vector_length)
    index.search(queries, 1)

Possibly related: https://github.com/facebookresearch/faiss/issues/2167.

1.7.4 worked fine. We only tried to upgrade to 1.8.0.post1 to get Python 3.12 support.

Platform

Macbook Pro M2 Max (2023). faiss installed using poetry from pypi.

OS:

Faiss version: 1.8.0.post1

Installed from: pypi (using poetry). Tested on both python 3.11 and 3.12.

Running on:

• [x] CPU
• [ ] GPU

Interface:

• [ ] C++
• [x] Python

Reproduction instructions

See above snippet.

[mengdilin]

The supported way of installing faiss is via conda. Have you tried using that (https://anaconda.org/pytorch/faiss-cpu) and see if that still produces on Mac M2?

[adampauls]

I tried installing with conda, and it does work. Unfortunately, using conda is not an option for us. Do you have any idea what changed between 1.7.4 and 1.8.0.post1 to break the pypi/pip installation? I thought perhaps brew install libomp would fix the issue, but it didn't.

[adampauls]

Possibly this?

[adampauls]

I think I know what's going on: faiss distributes a libomp.dylib, but so does torch. If I set the env variable KMP_DUPLICATE_LIB_OK=TRUE, then the crash goes away. Is it possible that before 1.8.0, libomp was not distributed in the pypi whl and so this all worked okay?

[adampauls]

I'll note that I did not try installing both torch and faiss-cpu in the same conda env. It's possible that that is broken as well.

[mengdilin]

We actually do not maintain pypi distribution so I'm not clear what changed, but it looks like there is a workaround for this issue for now