search

openai / human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"
MIT License
2.31k stars 330 forks source link

Error in canonical solution and tests for HumanEval/163 #20

Open bmosaicml opened 1 year ago

bmosaicml commented 1 year ago

The following solution and tests don't seem to match the description. Can you explain why we are bounding lower and upper to be between 2 and 8?

`def generate_integers(a, b): """ Given two positive integers a and b, return the even digits between a and b, in ascending order.

For example:
generate_integers(2, 8) => [2, 4, 6, 8]
generate_integers(8, 2) => [2, 4, 6, 8]
generate_integers(10, 14) => []
"""

lower = max(2, min(a, b))
upper = min(8, max(a, b))

return [i for i in range(lower, upper+1) if i % 2 == 0]

def check(candidate):

# Check some simple cases
assert candidate(2, 10) == [2, 4, 6, 8], "Test 1"
assert candidate(10, 2) == [2, 4, 6, 8], "Test 2"
assert candidate(132, 2) == [2, 4, 6, 8], "Test 3"
assert candidate(17,89) == [], "Test 4"`
iCSawyer commented 10 months ago

The natural language description in the docstring requires the program to find digits instead of numbers, which means a and b must be in [0, 9], and 2 and 8 are the largest and smallest digits. I think maybe this is why.