Tool: BEIR

[konabuta]

BEIR

Beir is a benchmark of information retrieval. It was used for benchmarking Azure AI Search at Microsoft tech community - Azure AI Search: Outperforming vector search with hybrid retrieval and ranking capabilities. So I would like to try it on.

Reference

source: https://github.com/beir-cellar/beir wiki: https://github.com/beir-cellar/beir/wiki

[konabuta]

Started to run a notebook - https://colab.research.google.com/drive/1HfutiEhHMJLXiWGT8pcipxT5L2TpYEdt?usp=sharing.

Found list of dataset available - https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/.

nq, scidocs, touche2020, arguana, climate-fever, dbpedia, fever, hotpotqa and covid were evaluated in the blog of Azure AI Search.

[konabuta]

Several models are evaluated. The results are as follows.

• msmarco-distilbert-base-v4

• msmarco-roberta-base-ance-firstp

• Elasticsearch (BM25)

• cross-encoder/ms-marco-electra-base

[konabuta]

Examples and tutorials - wiki page provides sample code for the following scenario.

• Lexical Retrieval: traditional text search using BM25.
• Dense Retrieval: embedding (dense vector) using Transformer models etc.
• Sparse Retrieval: embedding (sparse vector) using SPARTA, Pyserini, docT5query, DeepCT etc.
• Reranking: SBERT CE etc. etc