capreolus-ir / capreolus

A toolkit for end-to-end neural ad hoc retrieval
https://capreolus.ai
Apache License 2.0
95 stars 32 forks source link

Antique & metric-based relevance level & add relevance level in training data preparation #62

Closed crystina-z closed 4 years ago

crystina-z commented 4 years ago

(a rebased version of pr #61)

  1. support different relevance level for different metric at evaluation stage
  2. add relevance level for training data preparation
  3. add drmmtks reranker (reference here), which is used in antique

command to get map=0.175:

capreolus rerank.traineval with \
    benchmark.name=antique \
    benchmark.collection.name=antique \
    reranker.name=DRMM \
    rank.searcher.name=BM25Grid \
    rank.searcher.hits=100 \
    reranker.extractor.maxqlen=10 reranker.extractor.maxdoclen=60

TODO:

  1. configurable relevance level for dev evaluation?