faiss_gpu object is not linked to static library libfaiss.a

[Di-Is]

Summary

When building faiss_gpu, the objects of faiss_gpu are not copied to the static library libfaiss.a. Below is the static library capacity output to the build directory. Since the size of libfaiss.a is smaller than libfaiss_gpu.a, it is assumed that the objects of faiss_gpu are not copied to libfaiss.a.

-rw-r--r-- 1 root root 89M Apr 18 13:31 build/faiss/gpu/libfaiss_gpu.a
-rw-r--r-- 1 root root 11M Apr 18 13:32 build/faiss/libfaiss.a

In faiss/gpu/CMakeLists.txt#L292-L294 There is a description that links the faiss_gpu object to libfaiss.a, but it does not seem to be working.

target_link_libraries(faiss PRIVATE  "$<LINK_LIBRARY:WHOLE_ARCHIVE,faiss_gpu>")
target_link_libraries(faiss_avx2 PRIVATE "$<LINK_LIBRARY:WHOLE_ARCHIVE,faiss_gpu>")
target_link_libraries(faiss_avx512 PRIVATE "$<LINK_LIBRARY:WHOLE_ARCHIVE,faiss_gpu>")

Platform

OS:

• Host: Ubuntu22.04
• Container: Centos7(using quay.io/pypa/manylinux2014_x86_64:2024-01-23-12ffabc image)

Faiss version: 1.8.0

Installed from: src

Faiss compilation options:

• -DFAISS_ENABLE_GPU=ON
• -DFAISS_ENABLE_PYTHON=OFF
• -DBUILD_TESTING=OFF
• -DFAISS_OPT_LEVEL="generic"
• -DCMAKE_BUILD_TYPE=Release
• -DCMAKE_CUDA_ARCHITECTURES=80-real

Running on:

• [x] CPU
• [x] GPU

Interface:

• [X] C++
• [ ] Python

Reproduction instructions

The build and library installation commands are shown below.

# Install command
cmake . \
    -B build \
    -DFAISS_ENABLE_GPU=ON \
    -DFAISS_ENABLE_PYTHON=OFF \
    -DBUILD_TESTING=OFF \
    -DFAISS_OPT_LEVEL="generic" \
    -DCMAKE_CUDA_ARCHITECTURES="80-real" \
    -DCMAKE_BUILD_TYPE=Release && \
cmake --build build --config Release -j12 && \
cmake --install build

Check for static library size.

ls -lha faiss/*.a faiss/gpu/*.a
-rw-r--r-- 1 root root 89M Apr 18 13:31 build/faiss/gpu/libfaiss_gpu.a
-rw-r--r-- 1 root root 11M Apr 18 13:32 build/faiss/libfaiss.a

[mdouze]

This seems normal to me, libfaiss.a works standalone and contains only CPU code, libfaiss_gpu.a depends on the former and contains GPU code.

[Di-Is]

@mdouze Thanks for the reply! In FAISS v1.7.4, libfaiss_gpu.a was not generated and libfaiss.a contained both CPU and GPU objects. Has the specification changed in FAISS v1.8.0?

Also, it appears that libfaiss_gpu.a is not installed as described in issue #3375. This specification seems to make it difficult for users to use libfaiss_gpu.a.

[algoriddle]

libfaiss_gpu.a is supposed to be linked into libfaiss via the WHOLE_ARCHIVE directive in its entirety. Maybe this doesn't work when libfaiss is compiled as a static lib.

In FAISS v1.7.4, libfaiss_gpu.a was not generated and libfaiss.a contained both CPU and GPU objects. Has the specification changed in FAISS v1.8.0?

It's still the case that libfaiss*.so contains both CPU and GPU code. libfaiss_gpu.a was introduced to share the same objects for GPU code across the different versions of libfaiss (generic, avx2, avs512). This considerably reduced the build times for the conda packages. However, it's possible when built as a static library (libfaiss.a) that WHOLE_ARCHIVE packaging (of libfaiss_gpu into libfaiss) doesn't work.