search

castorini / rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.
http://rankllm.ai
Apache License 2.0
365 stars 43 forks source link

Easy way to run models on a list of examples? #117

Open orionw opened 6 months ago

orionw commented 6 months ago

Hi there! Great work on these -- would love to try them out!

I was curious if there is already a method/process for reranking given a list of pairs -- it seems like I would need to build out a Request object with candidates, but I'm not 100% certain.

Is there any minimal example scripts? Or perhaps one that loads from a JSON file that I can emulate?

Currently the README only has the one example that is fairly non-intuitive for new data that isn't an existing TREC dataset: I have to dig through several layers to find out what the inputs and outputs of these functions are and it still isn't quite clear how I could load my own data into it.

Any help would be greatly appreciated and I'm sure would be useful to the broader community as well!

TaKneeAa commented 6 months ago

Following!

sahel-sh commented 3 months ago

You are right, you need to create the request object with query and candidates. But alternatively you can create it from a json (see demo/rerank_inline_hits.py. You can also load retrieved results from a json file (see demo/rerank_stored_retrieved_results.py)