Closed lintool closed 2 years ago
We used to put ltr as a module under search.
But running ltr requires local Pyserini instead of pypi, so we put 'main' program for end-to-end runs back as a script, i.e. calling python scripts/ltr.py ...
.
Should we now change it to python -m pyserini.search.lucene.ltr ...
Though I think ibm reranker is fine? @yuki617
But running ltr requires local Pyserini instead of pypi, ...
What is there about the impl that requires a local installation? If it's the model, we can put under PYSERINI_CACHE, just like the indexes?
Ref #967
@stephaniewhoo anything more you want to do, or are we done here? if done, please go ahead and close issue.
I am good for now.
The IBM Model 1 reranker works (and is tested, yay!) - but the organization could be improved.
For example, there's no counterpart of a
Searcher
just "does the ranking":So, we have have two parts - a well-abstracted searcher, and a "main" program for end-to-end runs.
TranslationProbabilitySearcher
is probably the most descriptive name? IBM Model 1 is how to get the translation probabilities. Query likelihood and ColBERT's MaxSim is how we actually do the scoring.Ultimately, we want to be able to call something like
python -m pyserini.search.lucene.tprob ...
(tprob
is just my proposal for now).And the
ltr
stuff can maybe becomepython -m pyserini.search.lucene.ltr ...
This proposal will fit into the same structure as this: https://github.com/castorini/pyserini/issues/659#issuecomment-934308746