Failure to serialize in joblib

[NikhilSehgal123]

Describe the bug Consistently getting the following error when installing the faiss-cpu==1.7.3 module on Mac OS (Silicon M2 Chip): ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

To Reproduce pip install faiss-cpu==1.7.3

Expected behavior When installing the package, it is expected that the package scripts should include a script called:

swigfaiss_avx2.py

But this is missing when trying to install the module on my mac and I think this is causing the issue

Desktop (please complete the following information):

• OS: Mac
• Architecture: arm64
• Python: 3.9.16
• Version: v1.7.3

Additional context This happens when trying to "joblib.load()"

There are no avx2 CPU instructions in arm64 arch.

[NikhilSehgal123]

@kyamagu does this mean it is not possible right now to use this lib on arm64 computers?

[kyamagu]

Binary wheels are available. https://pypi.org/project/faiss-cpu/#files

Please post the exact crash log if anything is not working except showing ignorable error messages.

[NikhilSehgal123]

@kyamagu I tried a bunch of wheels but still getting an error:

pip install faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl

ERROR: faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl is not a supported wheel on this platform. (env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl ERROR: faiss_cpu-1.7.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform. (env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp310-cp310-macosx_11_0_arm64.whl ERROR: faiss_cpu-1.7.3-cp310-cp310-macosx_11_0_arm64.whl is not a supported wheel on this platform. (env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl ERROR: faiss_cpu-1.7.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.

[kyamagu]

Just a quick check. Are you using x86_64 CPython or arm64 CPython on Apple Silicon?

[NikhilSehgal123]

@kyamagu using arm64 CPython on Apple Silicon

[kyamagu]

Ok, this will need to investigate why the arch match is not working. Add verbose flag

ERROR: faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl is not a supported wheel on this platform.

[NikhilSehgal123]

ok thanks @kyamagu, let me know

[kyamagu]

@NikhilSehgal123 You've reported your env is the following

OS: Mac Architecture: arm64 Python: 3.9.16 Version: v1.7.3

You have to use faiss_cpu-1.7.3-cp39-cp39-macosx_11_0_arm64.whl. Not a cp311 wheel.

[NikhilSehgal123]

@kyamagu this is now installing correctly, but I'm still getting the error: ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

When inspecting the package lib, I still see the script "swigfaiss_avx2.py" is missing.

On my other Intel Mac, I can see it successfully installs this script

[kyamagu]

Is the error message causing a crash? Or, is the program only reporting the error message? Post your log.

[NikhilSehgal123]

@kyamagu my log below:

(env) (base) admin@Nikhils-MBP neo-agent % /Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/bin/python /Users/admin/Desktop/Repos/ChatMSP/neo-agent/neo_agent.py Traceback (most recent call last): File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/neo_agent.py", line 32, in vectorscore_woodland_trust = joblib.load('runbooks/vectorstore_woodland_trust.pkl') File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 605, in load obj = _unpickle(fobj, filename, mmap_mode) File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle obj = unpickler.load() File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1212, in load dispatchkey[0] File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1537, in load_stack_global self.append(self.find_class(module, name)) File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1579, in find_class import(module, level=0) ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

Looks like the "swigfaiss_avx2.py" script is not included in the package. If i check another PC, it works

[kyamagu]

I do not know what joblib is doing underneath, but I do not think any faiss component is picklable. If you are including a faiss index in the pickle, use faiss.write_index(). https://github.com/facebookresearch/faiss/issues/421

[NikhilSehgal123]

@kyamagu that is strange because my script is running on virtual machines which seem to be ok. I will look into this further

[NikhilSehgal123]

Hi @kyamagu when running my script on a i386 arch, it works perfectly. The main difference I'm noticing is what happens when I run

pip install faiss-cpu

On the i386 arch, I see all the python scripts installed including a script called swigfaiss_avx2.py. However on the arm64 arch, this script is missing when you run the pip install.

Can you confirm if this is expected? When you run the pip install, do you also see a script called swigfaiss_avx2.py?

[kyamagu]

avx2 module is not available in arm64 wheel, because it is not possible to compile.

[NikhilSehgal123]

@kyamagu understood, thank you. I'll close this thread with the understanding this is not possible on arm64 arch