search

lmarena / arena-hard-auto

Arena-Hard-Auto: An automatic LLM benchmark.
Apache License 2.0
606 stars 71 forks source link

Conv should be defined within choice loop #39

Closed zankner closed 1 month ago

zankner commented 2 months ago

My understanding is that num_choices corresponds to the number of different responses to an example we want to sample. When conv is defined it is done outside the loop for choices (https://github.com/lm-sys/arena-hard-auto/blob/6c8ad1ab0478ec3b53b8efaeffed71bcf0f8d75d/gen_answer.py#L49). So for each choice you actually build the previous answers across choices into conv. I think it makes sense to define conv inside the loop for each choice.

Sorry if I have wrong impression for what num_choices is

CodingWithTim commented 2 months ago

@zankner Thanks for bring this up! Would you be able to submit a PR to fix this issue? Appreciate your contribution!