Support Multi-node evaluation

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

http://beir.ai

Apache License 2.0

1.49k stars 177 forks source link

Open NouamaneTazi opened 11 months ago

NouamaneTazi commented 11 months ago

You can try this PR using:

torchrun --nproc_per_node=2 examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py

Using e5-large model I got

MSMARCO (8.84M documents) took 1h03min to encode (on 16 GPUs) -> evaluation took 1h04min
NQ (2.68M documents) took 22min to encode (on 16 GPUs) -> evaluation took 25min

cc @thakur-nandan