search

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
30.55k stars 3.56k forks source link

Tests failing on TestGpuIndexIVFPQ.LargeBatch #3068

Open weiwch opened 1 year ago

weiwch commented 1 year ago

Summary

TestGpuIndexIVFPQ.LargeBatch fails with NVIDIA GeForce RTX 3060 (12 GB vRAM) and CUDA 11.7.

Platform

OS: Ubuntu 20.04.6 LTS

Faiss version: d85601d972af2d64103769ab8d940db28aaae2a0

Installed from: compilation

Faiss compilation options: 

cmake -B build . -DFAISS_ENABLE_GPU=ON -DFAISS_ENABLE_PYTHON=OFF -DBUILD_TESTING=ON -DBUILD_SHARED_LIBS=ON -DCUDAToolkit_ROOT=/usr/local/cuda-11.7 -DCMAKE_CUDA_ARCHITECTURES="75"

(Compilation without MKL, with CUDA 11.7)

Running on:

Interface:

Reproduction instructions

Clone the repo and execute the following commands.

cmake -B build . -DFAISS_ENABLE_GPU=ON -DFAISS_ENABLE_PYTHON=OFF -DBUILD_TESTING=ON -DBUILD_SHARED_LIBS=ON -DCUDAToolkit_ROOT=/usr/local/cuda-11.7 -DCMAKE_CUDA_ARCHITECTURES="75"
make -C build -j faiss
make -C build -j 
sudo make -C build install

Tests will fails on TestGpuIndexIVFPQ.LargeBatch.

The following tests FAILED:
        153 - TestGpuIndexIVFPQ.LargeBatch (Failed)
Errors while running CTest

Here is the log (./build/Testing/Temporary/LastTest.log) about this error.


153/196 Testing: TestGpuIndexIVFPQ.LargeBatch
153/196 Test: TestGpuIndexIVFPQ.LargeBatch
Command: "/home/wwq/ANN/faiss/build/faiss/gpu/test/TestGpuIndexIVFPQ" "--gtest_filter=TestGpuIndexIVFPQ.LargeBatch" "--gtest_also_run_disabled_tests"
Directory: /home/wwq/ANN/faiss/build/faiss/gpu/test
"TestGpuIndexIVFPQ.LargeBatch" start time: Sep 16 12:20 CST
Output:
----------------------------------------------------------
testing with random seed 100
Note: Google Test filter = TestGpuIndexIVFPQ.LargeBatch
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from TestGpuIndexIVFPQ
[ RUN      ] TestGpuIndexIVFPQ.LargeBatch
WARNING clustering 2160 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2160 points to 256 centroids: please provide at least 9984 training points
Config
----------------------------
IVFPQ device 0 numVecs 2999 dim 4 numCentroids 54 codes 2 bitsPerCode 8 nprobe 54 numQuery 100000 k 7 usePrecomputed 1 indicesOpt 2 useFloat16 1
Result error and differences
----------------------------
max abs diff 0.0000011 rel diff 0.0055601
idx diff avg: 0.0018629 max: 1
idx diff of 1:      1304 (0.186% of queries)
idx diff of >1:     0 (0.000% of queries)
idx diff not found: 100 (0.014% of queries) [typically a last element inversion]
non-unique indices: 0 (a serious error if >0)
WARNING clustering 2360 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2360 points to 256 centroids: please provide at least 9984 training points
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.0572007 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (3, 10) refD: 0.021593540906906128 testD: 0.022865071892738342
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.0727348 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (3, 11) refD: 0.022865056991577148 testD: 0.024590905755758286
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.707001 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 1) refD: 0.0084939748048782349 testD: 0.017782852053642273
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.718276 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 2) refD: 0.0085610076785087585 testD: 0.018156155943870544
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.229584 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 3) refD: 0.015567265450954437 testD: 0.019604720175266266
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.749847 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 4) refD: 0.016933590173721313 testD: 0.037247270345687866
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.729431 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 5) refD: 0.017782852053642273 testD: 0.038201063871383667
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.760993 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 6) refD: 0.018156185746192932 testD: 0.040459118783473969
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.717886 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 7) refD: 0.01960473507642746 testD: 0.041559025645256042
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.540444 vs 0.035
IVFPQ device 0 numVecs 3516 dim 4 numCentroids 59 codes 2 bitsPerCode 8 nprobe 59 numQuery 100000 k 12 usePrecomputed 1 indicesOpt 0 useFloat16 1 (6, 8) refD: 0.02498268336057663 testD: 0.043483853340148926
/home/wwq/ANN/faiss/faiss/gpu/test/TestUtils.cpp:265: Failure
Expected: (relErr) <= (maxRelativeError), actual: 0.169567 vs 0.035
... (About 200,000 lines omitted)

All omitted lines are error reports about relative errors.

eres313 commented 3 months ago

any updates about this error?