search

castorini / rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.
http://rankllm.ai
Apache License 2.0
312 stars 39 forks source link

P2- Add variable window size support to rank_gpt class #43

Closed sahel-sh closed 8 months ago

sahel-sh commented 8 months ago

Currently only rank_listwise_os_llm.py supports variable window sizes. We should add this to rank_gpt as well.

sahel-sh commented 8 months ago

RankListwiseOSLLM uses the window size to have a better estimate of the number of output tokens which is in turn needed to determine the amount of passage truncation during prompt generation. We can change the RankGPT class to do the same. However the original implementation of rankgput does not have this feature and given that the GPT3.5 and GPT4 responses are not always in the requested format, this won't be an accurate estimate anyways.

AndreSlavescu commented 8 months ago

Working on this!

sahel-sh commented 8 months ago

@AndreSlavescu thank you for working on this so quickly, I think we can mark this issue as fixed now. Would you please confirm?

AndreSlavescu commented 8 months ago

Yes, it can be marked as complete.