Question about the decoding settings for alpaca_eval

allenai / open-instruct

Apache License 2.0

1.1k stars 145 forks source link

Question about the decoding settings for alpaca_eval #100

Closed huchinlp closed 5 months ago

huchinlp commented 6 months ago

Hi, according to the alpaca farm paper, they used sampling with a temperature setting of 0.7 for text generation. However, it appears your code utilizes greedy decoding (temperature=0), which could result in the generation of repetitive tokens and potential failure in the evaluation process (with alpaca_eval).

hamishivi commented 5 months ago

Hi, yeah, we decided to go for greedy decoding for our case and didn't tune generation parameters. Newer versions of alpaca eval should be able to handle this, and we are looking into updating alpaca eval accordingly!