search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.57k stars 191 forks source link

On the dependency `faiss` #71

Open jordane95 opened 2 years ago

jordane95 commented 2 years ago

Hi, I noticed that your beir==0.2.3 depends on faiss-cpu. While in your NeurIPS'2021 paper, you benchmarked several dense retrieval models on GPU. Did you use faiss-gpu for that? Have you considered update the dependency of beir to faiss-gpu?

thakur-nandan commented 2 years ago

Hi @jordane95,

We benchmark using exact-search and not faiss for the NeurIPS paper which used the GPU. You can check this example: https://github.com/UKPLab/beir/blob/main/examples/benchmarking/benchmark_sbert.py

Updating faiss-cpu to faiss-gpu could be interesting, however last time I checked we find issues in downloading faiss-gpu using PyPI. We decided to go with faiss-cpu for now for others to work on it easily. It will be interesting to integrate faiss-gpu in the future!

Kind regards, Nandan