Closed bcui19 closed 5 months ago
Hi, this is not a bug but an intended parameter for Arena Hard.
Since most of the prompts are difficult problem solving oriented questions (many of them being very technical), we use temperature 0 so models can produce more accurate answer. Further, we want to limit the variance of the win rate as well. Thanks.
The temperature for answer generations is always
0.0
, since thecategory
of all questions isarena-hard-v0.1.
This implicitly means that the temperature will be defaulted to0.0.
Wanted to flag since I think this might be undesirable behavior.