search

openai / human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"
MIT License
2.31k stars 330 forks source link

bug in estimate_pass_at_k #35

Open sidaw opened 8 months ago

sidaw commented 8 months ago

https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/evaluation.py#L26

This code returns 1 when c=0 and n < k, whereas 0 is expected.