search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.49k stars 177 forks source link

Typing for retriever is too strict: implement abstract retriever base class #152

Open jon-chuang opened 1 year ago

jon-chuang commented 1 year ago

One should be able to evaluate BEIR on other retrievers other than Union[Type[DRES], Type[DRFS], Type[BM25], Type[SS]

The work around is the use the static method evaluate. It's slightly awkward. See SPLADE's eval https://github.com/naver/splade/blob/06947de1da83e5443809f1b7e78937416d53c14b/splade/beir_eval.py#L85