Error in generate_leaderboard.py

boson-ai / RPBench-Auto

An automated pipeline for evaluating LLMs for role-playing.

Apache License 2.0

118 stars 3 forks source link

Error in generate_leaderboard.py #2

Open xuguozhi opened 3 weeks ago

xuguozhi commented 3 weeks ago

Exception has occurred: KeyError'winner' File "/home/RPBench-Auto/calculate_metrics.py", line 202, in get_metrics winner = extract_and_parse_json(obj["judger_response"])["winner"] File "/home/RPBench-Auto/generate_leaderboard.py", line 17, in <module> _, win_rate, model_list = get_metrics( KeyError: 'winner' 'eval_llama-3-1-8b-instruct_vs_gpt-4o.jsonl'

sxjscience commented 3 weeks ago

@xuguozhi Can you try the latest version?