Pyserini

Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. Retrieval using sparse representations is provided via integration with our group's Anserini IR toolkit, which is built on Lucene. Retrieval using dense representations is provided via integration with Facebook's Faiss library.

Pyserini is primarily designed to provide effective, reproducible, and easy-to-use first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, prebuilt indexes, and evaluation scripts for many commonly used IR test collections. With Pyserini, it's easy to reproduce runs on a number of standard IR test collections!

For additional details, our paper in SIGIR 2021 provides a nice overview.

✨ New! Guide to working with the MS MARCO 2.1 Document Corpus for TREC 2024 RAG Track.

❗ Anserini was upgraded from JDK 11 to JDK 21 at commit 272565 (2024/04/03), which corresponds to the release of v0.35.0. Correspondingly, Pyserini was upgraded to JDK 21 at commit b2f677 (2024/04/04).

🎬 Installation

Install via PyPI (requires Python 3.10+):

pip install pyserini

Sparse retrieval depends on Anserini, which is itself built on Lucene (written in Java), and thus requiring JDK 21.

Dense retrieval depends on neural networks and requires a more complex set of dependencies. A pip installation will automatically pull in the 🤗 Transformers library to satisfy the package requirements. Pyserini also depends on PyTorch and Faiss, but since these packages may require platform-specific custom configuration, they are not explicitly listed in the package requirements. We leave the installation of these packages to you.

The software ecosystem is rapidly evolving and a potential source of frustration is incompatibility among different versions of underlying dependencies. We provide additional detailed installation instructions here.

If you're planning on just using Pyserini, then the pip instructions above are fine. However, if you're planning on contributing to the codebase or want to work with the latest not-yet-released features, you'll need a development installation. Instructions are provided here.

🙋 How do I search?

Pyserini supports the following classes of retrieval models:

• Traditional lexical models (e.g., BM25) using LuceneSearcher.
• Learned sparse retrieval models (e.g., uniCOIL, SPLADE, etc.) using LuceneImpactSearcher.
• Learned dense retrieval models (e.g., DPR, Contriever, etc.) using FaissSearcher.
• Hybrid retrieval models (e.g., dense-sparse fusion) using HybridSearcher.

See this guide (same as the links above) for details on how to search common corpora in IR and NLP research (e.g., MS MARCO, NaturalQuestions, BEIR, etc.) using indexes that we have already built for you.

Once you get the top-k results, you'll actually want to fetch the document text... See this guide for how.

🙋 How do I index my own corpus?

Well, it depends on what type of retrieval model you want to search with:

• Building a BM25 Index (Direct Java Implementation)
• Building a BM25 Index (Embeddable Python Implementation)
• Building a Sparse Vector Index
• Building a Dense Vector Index

The steps are different for different classes of models: this guide (same as the links above) describes the details.

🙋 Additional FAQs

• How do I configure search? (Guide to Interactive Search)
• How do I manually download indexes? (Guide to Interactive Search)
• How do I perform dense and hybrid retrieval? (Guide to Interactive Search)
• How do I iterate over index terms and access term statistics? (Index Reader API)
• How do I traverse postings? (Index Reader API)
• How do I access and manipulate term vectors? (Index Reader API)
• How do I compute the tf-idf or BM25 score of a document? (Index Reader API)
• How do I access basic index statistics? (Index Reader API)
• How do I access underlying Lucene analyzers? (Analyzer API)
• How do I build custom Lucene queries? (Query Builder API)
• How do I iterate over raw collections? (Collection API)

⚗️ Reproducibility

With Pyserini, it's easy to reproduce runs on a number of standard IR test collections! We provide a number of prebuilt indexes that directly support reproducibility "out of the box".

In our SIGIR 2022 paper, we introduced "two-click reproductions" that allow anyone to reproduce experimental runs with only two clicks (i.e., copy and paste). Documentation is organized into reproduction matrices for different corpora that provide a summary of different experimental conditions and query sets:

• MS MARCO V1 Passage
• MS MARCO V1 Document
• MS MARCO V2 Passage
• MS MARCO V2 Document
• BEIR
• Mr.TyDi
• MIRACL
• Open-Domain Question Answering
• CIRAL

For more details, see our paper on Building a Culture of Reproducibility in Academic Research.

Additional reproduction guides below provide detailed step-by-step instructions.

📃 Additional Documentation

• Guide to prebuilt indexes
• Guide to interactive searching
• Guide to text classification with the 20Newsgroups dataset
• Guide to working with the COVID-19 Open Research Dataset (CORD-19)
• Guide to working with entity linking
• Guide to working with spaCy
• Usage of the Analyzer API
• Usage of the Index Reader API
• Usage of the Query Builder API
• Usage of the Collection API
• Direct Interaction via Pyjnius

📜️ Release History

• v0.36.0 (w/ Anserini v0.36.1): June 17, 2024 [Release Notes]
• v0.35.0 (w/ Anserini v0.35.0): April 4, 2024 [Release Notes]
• v0.25.0 (w/ Anserini v0.25.0): March 31, 2024 [Release Notes]
• v0.24.0 (w/ Anserini v0.24.0): December 28, 2023 [Release Notes]
• v0.23.0 (w/ Anserini v0.23.0): November 17, 2023 [Release Notes]
• v0.22.1 (w/ Anserini v0.22.1): October 19, 2023 [Release Notes]
• v0.22.0 (w/ Anserini v0.22.0): August 31, 2023 [Release Notes]
• v0.21.0 (w/ Anserini v0.21.0): April 6, 2023 [Release Notes]
• v0.20.0 (w/ Anserini v0.20.0): February 1, 2023 [Release Notes]

✨ References

If you use Pyserini, please cite the following paper:

@INPROCEEDINGS{Lin_etal_SIGIR2021_Pyserini,
   author = "Jimmy Lin and Xueguang Ma and Sheng-Chieh Lin and Jheng-Hong Yang and Ronak Pradeep and Rodrigo Nogueira",
   title = "{Pyserini}: A {Python} Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations",
   booktitle = "Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)",
   year = 2021,
   pages = "2356--2362",
}

🙏 Acknowledgments

This research is supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada.