My understanding is that num_choices corresponds to the number of different responses to an example we want to sample. When conv is defined it is done outside the loop for choices (https://github.com/lm-sys/arena-hard-auto/blob/6c8ad1ab0478ec3b53b8efaeffed71bcf0f8d75d/gen_answer.py#L49). So for each choice you actually build the previous answers across choices into conv. I think it makes sense to define conv inside the loop for each choice.
Sorry if I have wrong impression for what num_choices is
My understanding is that
num_choices
corresponds to the number of different responses to an example we want to sample. Whenconv
is defined it is done outside the loop for choices (https://github.com/lm-sys/arena-hard-auto/blob/6c8ad1ab0478ec3b53b8efaeffed71bcf0f8d75d/gen_answer.py#L49). So for each choice you actually build the previous answers across choices into conv. I think it makes sense to define conv inside the loop for each choice.Sorry if I have wrong impression for what num_choices is