search

18907305772 / FuseAI

FuseAI Project
https://huggingface.co/FuseAI
76 stars 34 forks source link

Questions about AlpacaEval 2.0 evaluation #23

Open lucasz05 opened 2 months ago

lucasz05 commented 2 months ago

Hello, we encountered some issues while reproducing the test results in the paper. On the AlpacaEval 2.0, we noticed that your GitHub page stated that you followed the default settings and chose 'alpaca_eval_gpt4_turbo-fn' for evaluation. However, we found that the default annotators_config of AlpacaEval 2.0 was 'weighted_alpaca_eval_gpt4_turbo'. Therefore, we conducted experiments using the above two as annotators respectively. The lc_winrate result for 'weighted_alpaca_eval_gpt4_turbo' was 15.88. And the lc_winrate result for 'alpaca_eval_gpt4_turbo-fn' was 22.42! Upon further examination, we found that the preference ratings in 'alpaca_eval_gpt4_turbo-fn' are 1.0, 2.0, and 1.5. According to the introduction on the alpaca_eval author's GitHub page, this is the scoring method for AlpacaEval 1.0. Therefore, we doubt whether there is an issue here. We would like to know what reference and annotator you use in your experiments.

lucasz05 commented 2 months ago

And we would like to know where you got the results about the Source LLMs in the paper, whether they were your experimental results or obtained from other places?