Open lmx760581375 opened 1 year ago
In fact, the prompts of humaneval-x and humaneval should be the same.
I have checked and found that my generated parameters are still different from yours, because I did not see the parameter configuration under readme due to my negligence.
When my temperature was set to 0 and the sampling strategy greedy sampling was turned on, My effect approximates your effect, my N=1 and everything is aligned according to your readme.
But I have a big question, does the fact that the parameters have such a large effect mean that the robustness is not good? The result of this test makes me feel that I can improve by 10 points just by adjusting the parameters, which feels exaggerated.
We have done three experiments:
From our results, the robustness of WizardCoder is good. Since the 59.8 score is from greedy decoding, this won't be affected by any randomness. If you cannot get this score, we believe your testing process must contain some problems. Since our models are evaluated with OpenAI's HumanEval, we suggest you follow our instructions to reproduce the results.
I downloaded the model weight of wizardcoder on huggingface, and tested N=200 on humaneval. The test script is your humaneval_gen.py. temperature=0.2 top_p=0.95 is the same as that mentioned in your paper, but the result I measured was only 47.