castorini / rank_llm

Repository for prompt-decoding using LLMs (GPT3.5, GPT4, Vicuna, and Zephyr)
http://rankllm.ai
Apache License 2.0
277 stars 36 forks source link

Fix response analysis #58

Closed sahel-sh closed 5 months ago

sahel-sh commented 5 months ago

This cl:

1- Merges the two versions of the analyze_response scripts, since the verbose one is mostly copy-pasted of the original one 2- It fixes the analysis to read from the new ranking_execution_summary folder 3- Instead of using the window size in the file name, for each prompt, it gets the proper number of passages based on the value in the prompt. This is because when the window size/stride and top_k_candidates don't align properly, the last window will have different number of passages in it.