openai / human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"
MIT License
2.31k stars 330 forks source link

Error in the prompt of HumanEval/47 #6

Open Kim-mins opened 3 years ago

Kim-mins commented 3 years ago
def median(l: list):
    """Return median of elements in the list l.
    >>> median([3, 1, 2, 4, 5])
    3
    >>> median([-10, 4, 6, 1000, 10, 20])
    15.0
    """

Maybe the output of median([-10, 4, 6, 1000, 10, 20]) should be 8 not 15.0 Can you please check it?

Here's the test with statistics library

>>> import statistics
>>> statistics.median([-10, 4, 6, 1000, 10, 20])
8.0
Magolor commented 1 year ago

I've found the same problem. However, the question is: Is this an error in the annotation process that needs to be fixed, or a feature intended to test the completion performance under imperfect prompts?