Adding dense retrieval support - Githubissues

castorini / chatty-goose

A Python framework for conversational search

38 stars 9 forks source link

Adding dense retrieval support #25

Open jacklin64 opened 3 years ago

jacklin64 commented 3 years ago

A starting point could be using pyserini' dense model for Msmarco first.

justram commented 3 years ago

Specifically, we have a bm25 searcher now: https://github.com/castorini/chatty-goose/blob/2047d4e3e3390609146859ec98f5eac167f07e47/chatty_goose/pipeline/retrieval_pipeline.py#L29

Need to support Pyserini's SimpleDenseSearcher: https://github.com/castorini/pyserini#dense-retrieval

To Do:

[ ] Encode CAsT passage corpus with Faiss (brute-force/hnsw). Ref: https://github.com/castorini/pyserini/blob/master/scripts/msmarco-passage/encode_corpus.py
[ ] Validate dense retrieval pipeline. Dense model candidate: TCT-ColBERT, ANCE, DPR