search

castorini / rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.
http://rankllm.ai
Apache License 2.0
312 stars 39 forks source link

Integrate result class in reranker #55

Closed sahel-sh closed 8 months ago

sahel-sh commented 8 months ago

This cl:

1- Changes all the reranking related classes to use Result and ResultsWriter 2- Instead of separate token_count and prompt_and_responses folders, the ranking exec summary is written in a single folder. 4- In addition to trec value format for the final rerank results, the sorted results file is also stored as a json 3- the evaluation script is fixed

Fixing response analysis script with come in a follow up cl TESTED=ran run_rank_llm e2e

sahel-sh commented 8 months ago

This cl has been waitng for review for two days. I will merge it now per our chat with @lintool to unblock followup cls needed for SIGIR2024 API release. @ronakice, @lintool if you have any suggestions, please file issues, and we can address them after the SIGIR release.

sahel-sh commented 8 months ago

This cl:

1- Changes all the reranking related classes to use Result and ResultsWriter 2- Instead of separate token_count and prompt_and_responses folders, the ranking exec summary is written in a single folder. 4- In addition to trec value format for the final rerank results, the sorted results file is also stored as a json 3- the evaluation script is fixed

Fixing response analysis script with come in a follow up cl TESTED=ran run_rank_llm e2e