search

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
31.23k stars 3.63k forks source link

call ivf search coredump through multithreads #3321

Closed shaozhixue closed 4 months ago

shaozhixue commented 7 months ago

Summary

my program call ivf's search coredump through multithreads. if the parameter nprobe is the same in all requests, problem disappeared. single thread has not the problem.

Platform

centos 7 x86_64

Faiss version: 1.70

Installed from: compile source code

Running on: -CPU

Interface:

error information:

terminate called after throwing an instance of 'faiss::FaissException' what(): Error in faiss::IndexIVF::search_preassigned(faiss::Index::idx_t, const float, faiss::Index::idx_t, const idx_t, const float, float, faiss::Index::idx_t, bool, const faiss::IVFSearchParameters, faiss::IndexIVFStats*) const::__lambda4 at /data1/minisearch/upload/zhixue3/git/vs/third-64/faiss/faiss/IndexIVF.cpp:459: Error: 'key < (idx_t)nlist' failed: Invalid key=4537784286748968700 nlist=20480

I suspect this issue is related to OpenMP.

How should this issue be resolved?

alexanderguzhva commented 7 months ago

Do I get it correctly that you'd like to run index.search() using multiple threads, and search thread wants to use its own nprobe? If so, then you need to pass up const faiss::IVFSearchParameters* parameter, which allows to configure nprobe for each search request

shaozhixue commented 7 months ago

Do I get it correctly that you'd like to run index.search() using multiple threads, and search thread wants to use its own nprobe? If so, then you need to pass up const faiss::IVFSearchParameters* parameter, which allows to configure nprobe for each search request

Thank you for your reply, I will try it right away