Open alexanderguzhva opened 1 month ago
alexanderguzhva
commented
1 month ago Depending on the OS version in a docker contailer, compiler version, the results of the unit tests (what unit tests exactly fail) are different. I've tried multiple variants.
mdouze
commented
1 month ago Hmmm that's weird, we don't observe that.
alexanderguzhva
commented
1 month ago Well, let me build a docker container that reproduces this problem then
Charlyo
commented
3 weeks ago Any news regarding this? I have similar problems with faiss on bookworm debian.
victor-shepardson
commented
3 weeks ago I'm encountering a similar crash only when also importing torch on macOS 12.7.5 and Python 3.10.14.
Merely importing torch can precipitate the crash. Within pytest, I see a similar error as above:
tests/test_io.py .Fatal Python error: Aborted Current thread 0x0000000100f38580 (most recent call first): File "/Users/victor/mambaforge/envs/test-local/lib/python3.10/site-packages/faiss/swigfaiss.py", line 2277 in search File "/Users/victor/mambaforge/envs/test-local/lib/python3.10/site-packages/faiss/class_wrappers.py", line 343 in replacement_search ... Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, faiss._swigfaiss
in a different context, I sometimes get this warning:
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
torch (2.2.2) and faiss-cpu (1.8.0) dependencies are resolved together by poetry and installed from PyPI.
I tried upgrading torch to 2.3.1 (via PyPI) and it did not help.
However, installing torch and faiss-cpu from the pytorch channel with conda before using poetry install seems to prevent the crash.
mdouze
commented
2 weeks ago I prefer trying to repro this on docker + linux only. It is surprising that the CI does not catch this since it since it also uses docker and linux. The process crashes with sigabrt, which usually means that abort() was called due to some internal inconsistency. Usually, there is some error message displayed, but this does not seem to be the case here.
Charlyo
commented
2 weeks ago From what I was able to diagnose with debian 12, It seems that pytest seems to import incorrect libs (system vs precompiled within the created wheel). Haven't investigated more. @mnorris11 You might want to check this.
mdouze
commented
2 weeks ago @mnorris11 can you try to repro on a AWS machine?
alexanderguzhva
commented
1 week ago @mdouze
Dockerfile
FROM ubuntu:22.04
RUN apt update && apt install -y python3 python3-pip gcc g++ mc git swig sudo libomp-dev libopenblas-dev wget
RUN pip3 install numpy==1.26.4 scipy pytest
RUN cd /root && git clone https://github.com/facebookresearch/faiss
RUN wget -qO- "https://cmake.org/files/v3.26/cmake-3.26.5-linux-x86_64.tar.gz" | sudo tar --strip-components=1 -xz -C /usr/local
RUN cd /root/faiss && /usr/local/bin/cmake -B build -DFAISS_ENABLE_GPU=OFF -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Release .
RUN cd /root/faiss && make -C build -j 8 faiss
RUN cd /root/faiss && make -C build -j 8 swigfaiss
RUN cd /root/faiss/build/faiss/python && python3 setup.py install
RUN cd /root/faiss/tests && python3 -m pytestbuilding an image on an AWS machine, ubuntu 22.04
$ docker build --tag bad_faiss .
[+] Building 221.3s (13/13) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 741B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:22.04 0.0s
=> [ 1/10] FROM docker.io/library/ubuntu:22.04 0.0s
=> [ 2/10] RUN apt update && apt install -y python3 python3-pip gcc g++ mc git swig sudo libomp-dev libopenblas-dev wget 52.5s
=> [ 3/10] RUN pip3 install numpy==1.26.4 scipy pytest 7.6s
=> [ 4/10] RUN cd /root && git clone https://github.com/facebookresearch/faiss 10.9s
=> [ 5/10] RUN wget -qO- "https://cmake.org/files/v3.26/cmake-3.26.5-linux-x86_64.tar.gz" | sudo tar --strip-components=1 -x 6.2s
=> [ 6/10] RUN cd /root/faiss && /usr/local/bin/cmake -B build -DFAISS_ENABLE_GPU=OFF -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE= 2.7s
=> [ 7/10] RUN cd /root/faiss && make -C build -j 8 faiss 73.0s
=> [ 8/10] RUN cd /root/faiss && make -C build -j 8 swigfaiss 64.2s
=> [ 9/10] RUN cd /root/faiss/build/faiss/python && python3 setup.py install 1.4s
=> ERROR [10/10] RUN cd /root/faiss/tests && python3 -m pytest 2.8s
------
> [10/10] RUN cd /root/faiss/tests && python3 -m pytest:
1.554 ============================= test session starts ==============================
1.554 platform linux -- Python 3.10.12, pytest-8.2.2, pluggy-1.5.0
1.554 rootdir: /root/faiss/tests
1.554 collected 849 items
1.554
1.554 test_autotune.py ..... [ 0%]
1.561 test_binary_factory.py ...... [ 1%]
1.564 test_binary_hashindex.py ...... [ 2%]
2.027 test_binary_io.py ...... [ 2%]
2.077 test_build_blocks.py .................................................. [ 8%]
2.668 test_callback_py.py . [ 8%]
2.686 test_clone.py Fatal Python error: Aborted
2.705
2.705 Current thread 0x0000796e61b891c0 (most recent call first):
2.705 File "/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/swigfaiss.py", line 4828 in train
2.705 File "/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/class_wrappers.py", line 298 in replacement_train
2.705 File "/root/faiss/tests/test_clone.py", line 28 in do_test_clone
2.705 File "/root/faiss/tests/test_clone.py", line 79 in test_AdditiveQuantizer
2.705 File "/usr/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
2.705 File "/usr/lib/python3.10/unittest/case.py", line 591 in run
2.705 File "/usr/lib/python3.10/unittest/case.py", line 650 in __call__
2.705 File "/usr/local/lib/python3.10/dist-packages/_pytest/unittest.py", line 344 in runtest
2.705 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 173 in pytest_runtest_call
2.705 File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
2.705 File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2.706 File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 241 in <lambda>
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 341 in from_call
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 240 in call_and_report
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 135 in runtestprotocol
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
2.706 File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
2.706 File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2.706 File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
2.706 File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 364 in pytest_runtestloop
2.706 File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
2.707 File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2.707 File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
2.707 File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 339 in _main
2.707 File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 285 in wrap_session
2.707 File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 332 in pytest_cmdline_main
2.707 File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
2.707 File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
2.707 File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
2.707 File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 178 in main
2.707 File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 206 in console_main
2.707 File "/usr/local/lib/python3.10/dist-packages/pytest/__main__.py", line 7 in <module>
2.707 File "/usr/lib/python3.10/runpy.py", line 86 in _run_code
2.707 File "/usr/lib/python3.10/runpy.py", line 196 in _run_module_as_main
2.708
2.708 Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, faiss._swigfaiss, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation (total: 53)
2.781 Aborted (core dumped)
------
Dockerfile:11
--------------------
9 | RUN cd /root/faiss && make -C build -j 8 swigfaiss
10 | RUN cd /root/faiss/build/faiss/python && python3 setup.py install
11 | >>> RUN cd /root/faiss/tests && python3 -m pytest
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /root/faiss/tests && python3 -m pytest" did not complete successfully: exit code: 134AWS system
$ uname -a
Linux ip-172-31-0-56 6.5.0-1020-aws #20~22.04.1-Ubuntu SMP Wed May 1 16:10:50 UTC 2024 x86_64 x86_64 x86_64 GNU/Linuxbuilding an image on my linux laptop. clone test seems to pass (although, I had an impression that I had problems with tests on my laptop as well... maybe, this is a Schroedinger's bug), but two other tests fail. Again, this was not captured by standard facebook CI tests.
Step 10/10 : RUN cd /root/faiss/tests && python3 -m pytest
---> Running in 84080ddadb3e
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.2.2, pluggy-1.5.0
rootdir: /root/faiss/tests
collected 849 items
test_autotune.py ..... [ 0%]
test_binary_factory.py ...... [ 1%]
test_binary_hashindex.py ...... [ 2%]
test_binary_io.py ...... [ 2%]
test_build_blocks.py .................................................. [ 8%]
test_callback_py.py . [ 8%]
test_clone.py ........... [ 10%]
test_clustering.py ............ [ 11%]
test_contrib.py ......F.......................... [ 15%]
test_contrib_with_scipy.py ... [ 15%]
test_documentation.py . [ 15%]
test_doxygen_documentation.py . [ 15%]
test_extra_distances.py ............ [ 17%]
test_factory.py ................................... [ 21%]
test_fast_scan.py .s.................................................... [ 27%]
........... [ 29%]
test_fast_scan_ivf.py .................................................. [ 34%]
........................................................................ [ 43%]
........................................................................ [ 51%]
. [ 52%]
test_graph_based.py ...................... [ 54%]
test_index.py .................................. [ 58%]
test_index_accuracy.py .............................. [ 62%]
test_index_binary.py ................ [ 64%]
test_index_binary_from_float.py ..... [ 64%]
test_index_composite.py ....................................... [ 69%]
test_io.py .............. [ 70%]
test_ivflib.py ..... [ 71%]
test_local_search_quantizer.py ......F.................. [ 74%]
test_merge_index.py ................. [ 76%]
test_meta_index.py ..... [ 77%]
test_omp_threads_py.py . [ 77%]
test_oom_exception.py . [ 77%]
test_partition.py ................................................... [ 83%]
test_product_quantizer.py ............. [ 84%]
test_referenced_objects.py ......... [ 85%]
test_refine.py ........... [ 87%]
test_residual_quantizer.py ....................................... [ 91%]
test_rowwise_minmax.py .. [ 91%]
test_search_params.py ..................................... [ 96%]
test_standalone_codec.py ............................... [100%]
=================================== FAILURES ===================================
___________________ TestExhaustiveSearch.test_query_iterator ___________________
self = <test_contrib.TestExhaustiveSearch testMethod=test_query_iterator>
metric = 1
def test_query_iterator(self, metric=faiss.METRIC_L2):
ds = datasets.SyntheticDataset(32, 0, 1000, 1000)
xq = ds.get_queries()
xb = ds.get_database()
D, I = faiss.knn(xq, xb, 10, metric=metric)
threshold = float(D[:, -1].mean())
index = faiss.IndexFlat(32, metric)
index.add(xb)
ref_lims, ref_D, ref_I = index.range_search(xq, threshold)
def matrix_iterator(xb, bs):
for i0 in range(0, xb.shape[0], bs):
yield xb[i0:i0 + bs]
# check repro OK
_, new_lims, new_D, new_I = range_search_max_results(
index, matrix_iterator(xq, 100), threshold, max_results=1e10)
evaluation.check_ref_range_results(
ref_lims, ref_D, ref_I,
new_lims, new_D, new_I
)
max_res = ref_lims[-1] // 2
new_threshold, new_lims, new_D, new_I = range_search_max_results(
index, matrix_iterator(xq, 100), threshold, max_results=max_res)
self.assertLessEqual(new_lims[-1], max_res)
ref_lims, ref_D, ref_I = index.range_search(xq, new_threshold)
> evaluation.check_ref_range_results(
ref_lims, ref_D, ref_I,
new_lims, new_D, new_I
)
test_contrib.py:177:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/contrib/evaluation.py:269: in check_ref_range_results
np.testing.assert_array_equal(Lref, Lnew)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (<built-in function eq>, array([ 0, 14, 17, ..., 4789, 4794, 4798], dtype=uint64), array([ 0, 14, 17, ..., 4788, 4793, 4797], dtype=uint64))
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
@wraps(func)
def inner(*args, **kwds):
with self._recreate_cm():
> return func(*args, **kwds)
E AssertionError:
E Arrays are not equal
E
E Mismatched elements: 564 / 1001 (56.3%)
E Max absolute difference: 1
E Max relative difference: 0.00051046
E x: array([ 0, 14, 17, ..., 4789, 4794, 4798], dtype=uint64)
E y: array([ 0, 14, 17, ..., 4788, 4793, 4797], dtype=uint64)
/usr/lib/python3.10/contextlib.py:79: AssertionError
_______________ TestComponents.test_update_codebooks_with_double _______________
self = <test_local_search_quantizer.TestComponents testMethod=test_update_codebooks_with_double>
def test_update_codebooks_with_double(self):
"""If the data is not zero-centering, it would be more accurate to
use double-precision floating-point numbers."""
ds = datasets.SyntheticDataset(16, 1000, 1000, 0)
xt = ds.get_train() + 1000
xb = ds.get_database() + 1000
M = 4
nbits = 4
lsq = faiss.LocalSearchQuantizer(ds.d, M, nbits)
lsq.train(xt)
err_double = eval_codec(lsq, xb)
lsq = faiss.LocalSearchQuantizer(ds.d, M, nbits)
lsq.update_codebooks_with_double = False
lsq.train(xt)
err_float = eval_codec(lsq, xb)
# 6533.377 vs 25457.99
> self.assertLess(err_double, err_float)
E AssertionError: 6374.6978 not less than 6140.4946
test_local_search_quantizer.py:199: AssertionError
=========================== short test summary info ============================
FAILED test_contrib.py::TestExhaustiveSearch::test_query_iterator - Assertion...
FAILED test_local_search_quantizer.py::TestComponents::test_update_codebooks_with_double
============= 2 failed, 846 passed, 1 skipped in 553.77s (0:09:13) =============
Training ResidualQuantizer, with 4 steps on 3000 32D vectors
[0.026 s, 0.022 s clustering] train stage 0, 6 bits, kmeans objective 33217.3, total distance 200231, beam_size 1->5 (batch size 3000)
[0.160 s, 0.118 s clustering] train stage 1, 6 bits, kmeans objective 155050, total distance 143425, beam_size 5->5 (batch size 3000)
[0.243 s, 0.186 s clustering] train stage 2, 6 bits, kmeans objective 112965, total distance 107746, beam_size 5->5 (batch size 3000)
[0.324 s, 0.252 s clustering] train stage 3, 6 bits, kmeans objective 85549.6, total distance 81995.9, beam_size 5->5 (batch size 3000)
[0.002 s] encode stage 0, 6 bits, total error 67433, beam_size 5
[0.007 s] encode stage 1, 6 bits, total error 51657.3, beam_size 5
[0.013 s] encode stage 2, 6 bits, total error 40562.2, beam_size 5
[0.018 s] encode stage 3, 6 bits, total error 32055.1, beam_size 5
The command '/bin/sh -c cd /root/faiss/tests && python3 -m pytest' returned a non-zero code: 1
mnorris11
commented
1 week ago Also reproduced some sort of failure on an AWS instance:
# uname -a
Linux ip-172-31-61-48 6.5.0-1017-aws #17~22.04.2-Ubuntu SMP Mon Mar 25 20:28:54 UTC 2024 x86_64 x86_64 x86_64 GNU/LinuxInstallation commands are the same as report:
# apt update && apt install python3 python3-pip gcc g++ mc git swig sudo libomp-dev libopenblas-dev
# pip3 install numpy scipy pytest
# git clone https://github.com/facebookresearch/faiss
# wget -qO- "https://cmake.org/files/v3.26/cmake-3.26.5-linux-x86_64.tar.gz" | sudo tar --strip-components=1 -xz -C /usr/local
# cd faiss
# /usr/local/bin/cmake -B build -DFAISS_ENABLE_GPU=OFF -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Release .
# make -C build -j 8 faiss
# make -C build -j 8 swigfaiss
# cd build/faiss/python && python3 setup.py install
note: used python3 here
# cd /home/ubuntu/faiss/tests
# python3 -m pytestTest results:
root@ip-172-31-61-48:/home/ubuntu/faiss/tests# python3 -m pytest
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.10.12, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/ubuntu/faiss/tests
collected 849 items
test_autotune.py ..... [ 0%]
test_binary_factory.py ...... [ 1%]
test_binary_hashindex.py ...... [ 2%]
test_binary_io.py ...... [ 2%]
test_build_blocks.py .................................................. [ 8%]
test_callback_py.py . [ 8%]
test_clone.py ........... [ 10%]
test_clustering.py ............ [ 11%]
test_contrib.py ......F.........F................ [ 15%]
test_contrib_with_scipy.py ... [ 15%]
test_documentation.py . [ 15%]
test_doxygen_documentation.py . [ 15%]
test_extra_distances.py ............ [ 17%]
test_factory.py ................................... [ 21%]
test_fast_scan.py .s............................................................... [ 29%]
test_fast_scan_ivf.py .................................................................................................................................................. [ 46%]
................................................. [ 52%]
test_graph_based.py ...................... [ 54%]
test_index.py .................................. [ 58%]
test_index_accuracy.py .............................. [ 62%]
test_index_binary.py ................ [ 64%]
test_index_binary_from_float.py ..... [ 64%]
test_index_composite.py ....................................... [ 69%]
test_io.py .............. [ 70%]
test_ivflib.py ..... [ 71%]
test_local_search_quantizer.py ......F.................. [ 74%]
test_merge_index.py ................. [ 76%]
test_meta_index.py ..... [ 77%]
test_omp_threads_py.py . [ 77%]
test_oom_exception.py . [ 77%]
test_partition.py ................................................... [ 83%]
test_product_quantizer.py ............. [ 84%]
test_referenced_objects.py ......... [ 85%]
test_refine.py ........... [ 87%]
test_residual_quantizer.py ....................................... [ 91%]
test_rowwise_minmax.py .. [ 91%]
test_search_params.py ..................................... [ 96%]
test_standalone_codec.py ............................... [100%]
=================================================================================== FAILURES ===================================================================================
___________________________________________________________________ TestExhaustiveSearch.test_query_iterator ___________________________________________________________________
self = <test_contrib.TestExhaustiveSearch testMethod=test_query_iterator>, metric = 1
def test_query_iterator(self, metric=faiss.METRIC_L2):
ds = datasets.SyntheticDataset(32, 0, 1000, 1000)
xq = ds.get_queries()
xb = ds.get_database()
D, I = faiss.knn(xq, xb, 10, metric=metric)
threshold = float(D[:, -1].mean())
index = faiss.IndexFlat(32, metric)
index.add(xb)
ref_lims, ref_D, ref_I = index.range_search(xq, threshold)
def matrix_iterator(xb, bs):
for i0 in range(0, xb.shape[0], bs):
yield xb[i0:i0 + bs]
# check repro OK
_, new_lims, new_D, new_I = range_search_max_results(
index, matrix_iterator(xq, 100), threshold, max_results=1e10)
evaluation.check_ref_range_results(
ref_lims, ref_D, ref_I,
new_lims, new_D, new_I
)
max_res = ref_lims[-1] // 2
new_threshold, new_lims, new_D, new_I = range_search_max_results(
index, matrix_iterator(xq, 100), threshold, max_results=max_res)
self.assertLessEqual(new_lims[-1], max_res)
ref_lims, ref_D, ref_I = index.range_search(xq, new_threshold)
> evaluation.check_ref_range_results(
ref_lims, ref_D, ref_I,
new_lims, new_D, new_I
)
test_contrib.py:177:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/contrib/evaluation.py:269: in check_ref_range_results
np.testing.assert_array_equal(Lref, Lnew)
/usr/local/lib/python3.10/dist-packages/numpy/_utils/__init__.py:85: in wrapper
return fun(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (<built-in function eq>, array([ 0, 14, 17, ..., 4789, 4794, 4798], dtype=uint64), array([ 0, 14, 17, ..., 4788, 4793, 4797], dtype=uint64))
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
@wraps(func)
def inner(*args, **kwds):
with self._recreate_cm():
> return func(*args, **kwds)
E AssertionError:
E Arrays are not equal
E
E Mismatched elements: 564 / 1001 (56.3%)
E Max absolute difference among violations: 1
E Max relative difference among violations: 0.00051046
E ACTUAL: array([ 0, 14, 17, ..., 4789, 4794, 4798], dtype=uint64)
E DESIRED: array([ 0, 14, 17, ..., 4788, 4793, 4797], dtype=uint64)
/usr/lib/python3.10/contextlib.py:79: AssertionError
__________________________________________________________________________ TestPreassigned.test_float __________________________________________________________________________
self = <test_contrib.TestPreassigned testMethod=test_float>
def test_float(self):
ds = datasets.SyntheticDataset(128, 2000, 2000, 200)
d = ds.d
xt = ds.get_train()
xq = ds.get_queries()
xb = ds.get_database()
# define alternative quantizer on the 20 first dims of vectors
km = faiss.Kmeans(20, 50)
km.train(xt[:, :20].copy())
alt_quantizer = km.index
index = faiss.index_factory(d, "IVF50,PQ16np")
index.by_residual = False
# (optional) fake coarse quantizer
fake_centroids = np.zeros((index.nlist, index.d), dtype="float32")
index.quantizer.add(fake_centroids)
# train the PQ part
index.train(xt)
# add elements xb
a = alt_quantizer.search(xb[:, :20].copy(), 1)[1].ravel()
ivf_tools.add_preassigned(index, xb, a)
# search elements xq, increase nprobe, check 4 first results w/
# groundtruth
prev_inter_perf = 0
for nprobe in 1, 10, 20:
index.nprobe = nprobe
a = alt_quantizer.search(xq[:, :20].copy(), index.nprobe)[1]
D, I = ivf_tools.search_preassigned(index, xq, 4, a)
inter_perf = faiss.eval_intersection(
I, ds.get_groundtruth()[:, :4])
self.assertTrue(inter_perf >= prev_inter_perf)
prev_inter_perf = inter_perf
# test range search
index.nprobe = 20
a = alt_quantizer.search(xq[:, :20].copy(), index.nprobe)[1]
# just to find a reasonable radius
D, I = ivf_tools.search_preassigned(index, xq, 4, a)
radius = D.max() * 1.01
> lims, DR, IR = ivf_tools.range_search_preassigned(index, xq, radius, a)
test_contrib.py:376:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/contrib/ivf_tools.py:85: in range_search_preassigned
index_ivf.range_search_preassigned_c(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <faiss.swigfaiss.IndexIVFPQ; proxy of <Swig Object of type 'faiss::IndexIVFPQ *' at 0x7c0632b82370> >, nx = 200
x = <Swig Object of type 'faiss::IndexReplicasTemplate< faiss::Index >::component_t *' at 0x7c0632b81020>, radius = np.float32(83.655495)
keys = <Swig Object of type 'int_fast16_t *' at 0x7c0632b803f0>
coarse_dis = <Swig Object of type 'faiss::IndexReplicasTemplate< faiss::Index >::component_t *' at 0x7c0632b82070>
result = <faiss.swigfaiss.RangeSearchResult; proxy of <Swig Object of type 'faiss::RangeSearchResult *' at 0x7c0632b83cf0> >, store_pairs = False, params = None, stats = None
def range_search_preassigned(self, nx, x, radius, keys, coarse_dis, result, store_pairs=False, params=None, stats=None):
> return _swigfaiss.IndexIVF_range_search_preassigned(self, nx, x, radius, keys, coarse_dis, result, store_pairs, params, stats)
E TypeError: Wrong number or type of arguments for overloaded function 'IndexIVF_range_search_preassigned'.
E Possible C/C++ prototypes are:
E faiss::IndexIVF::range_search_preassigned(faiss::idx_t,float const *,float,faiss::idx_t const *,float const *,faiss::RangeSearchResult *,bool,faiss::IVFSearchParameters const *,faiss::IndexIVFStats *) const
E faiss::IndexIVF::range_search_preassigned(faiss::idx_t,float const *,float,faiss::idx_t const *,float const *,faiss::RangeSearchResult *,bool,faiss::IVFSearchParameters const *) const
E faiss::IndexIVF::range_search_preassigned(faiss::idx_t,float const *,float,faiss::idx_t const *,float const *,faiss::RangeSearchResult *,bool) const
E faiss::IndexIVF::range_search_preassigned(faiss::idx_t,float const *,float,faiss::idx_t const *,float const *,faiss::RangeSearchResult *) const
/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/swigfaiss.py:6035: TypeError
----------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 2000 points to 256 centroids: please provide at least 9984 training points
_______________________________________________________________ TestComponents.test_update_codebooks_with_double _______________________________________________________________
self = <test_local_search_quantizer.TestComponents testMethod=test_update_codebooks_with_double>
def test_update_codebooks_with_double(self):
"""If the data is not zero-centering, it would be more accurate to
use double-precision floating-point numbers."""
ds = datasets.SyntheticDataset(16, 1000, 1000, 0)
xt = ds.get_train() + 1000
xb = ds.get_database() + 1000
M = 4
nbits = 4
lsq = faiss.LocalSearchQuantizer(ds.d, M, nbits)
lsq.train(xt)
err_double = eval_codec(lsq, xb)
lsq = faiss.LocalSearchQuantizer(ds.d, M, nbits)
lsq.update_codebooks_with_double = False
lsq.train(xt)
err_float = eval_codec(lsq, xb)
# 6533.377 vs 25457.99
> self.assertLess(err_double, err_float)
E AssertionError: np.float32(6374.6978) not less than np.float32(6140.4946)
test_local_search_quantizer.py:199: AssertionError
=============================================================================== warnings summary ===============================================================================
../../../../usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/loader.py:30
/usr/local/lib/python3.10/dist-packages/faiss-1.8.0-py3.10.egg/faiss/loader.py:30: DeprecationWarning: numpy.core._multiarray_umath is deprecated and has been renamed to numpy._core._multiarray_umath. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core._multiarray_umath.__cpu_features__.
from numpy.core._multiarray_umath import __cpu_features__
test_build_blocks.py::TestPCA::test_pca
/usr/lib/python3.10/unittest/case.py:1242: RuntimeWarning: overflow encountered in cast
if not a > b:
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================== short test summary info ============================================================================
FAILED test_contrib.py::TestExhaustiveSearch::test_query_iterator - AssertionError:
FAILED test_contrib.py::TestPreassigned::test_float - TypeError: Wrong number or type of arguments for overloaded function 'IndexIVF_range_search_preassigned'.
FAILED test_local_search_quantizer.py::TestComponents::test_update_codebooks_with_double - AssertionError: np.float32(6374.6978) not less than np.float32(6140.4946)
======================================================= 3 failed, 845 passed, 1 skipped, 2 warnings in 380.46s (0:06:20) =======================================================
Training ResidualQuantizer, with 4 steps on 3000 32D vectors
[0.084 s, 0.082 s clustering] train stage 0, 6 bits, kmeans objective 33217.3, total distance 200231, beam_size 1->5 (batch size 3000)
[0.157 s, 0.146 s clustering] train stage 1, 6 bits, kmeans objective 155050, total distance 143425, beam_size 5->5 (batch size 3000)
[0.209 s, 0.190 s clustering] train stage 2, 6 bits, kmeans objective 112965, total distance 107746, beam_size 5->5 (batch size 3000)
[0.259 s, 0.232 s clustering] train stage 3, 6 bits, kmeans objective 85549.6, total distance 81995.9, beam_size 5->5 (batch size 3000)
[0.001 s] encode stage 0, 6 bits, total error 67433, beam_size 5
[0.004 s] encode stage 1, 6 bits, total error 51657.3, beam_size 5
[0.007 s] encode stage 2, 6 bits, total error 40562.2, beam_size 5
[0.009 s] encode stage 3, 6 bits, total error 32055.1, beam_size 5Some package versions are different, so I'll see next if I can repro with the older versions on AWS. My server (working):
AWS (failing):
Edit: Even with the older packages, failure seems to repro. Will keep looking
mnorris11
commented
4 days ago Hi folks, was anyone able to repro with conda? I was not able to, after updating the packages to the same version as the pip ones.
Per the team, faiss does not officially support pip.
I will go ahead and close this in a few weeks if we cannot repro with conda.
Summary
unexpected crashes while running
python3 -m pytestin a docker containerPlatform
ubuntu 22.04, docker
standard g++
Faiss version: master (4d06d7069fca2359e5b56d33c762ab91d015ee9d)
Installed from: compiled on my own
Faiss compilation options:
Running on:
Interface:
Reproduction instructions
then, run
python3 -m pytestfromtestsdirectory.removing
test_clone.pywon't help, because there will be moooooore problemscompiling with avx2 won't help