Closed lhl closed 2 months ago
No, it's just for testing. We tested both system prompts. (the one shown on the Llama 3 hf repo or set to None)
Could you set the default to None on the main ?
@jmercat Done
@Psycoy It appears from your answer in #35 That the prompt used for the leaderboard results is actually the pirate talk one. I asked that no prompt should be the default on main thinking that it was the one to use to reproduce leaderboard results but since it is not, maybe main should be reverted back to what it was (see code line).
It seems like this seems would generate bad benchmark results?
https://github.com/Psycoy/MixEval/blob/main/mix_eval/models/llama_3_8b_instruct.py#L18C1-L18C160
Note this seems to be an issue for llama_3_70b_instruct.py and zephyr_7b_beta.py
Is this what was run to generate the scores on https://mixeval.github.io/#leaderboard ?!?!