search

openai / grade-school-math

1.05k stars 148 forks source link

Problem #455 has incorrect solution #9

Closed myclamm closed 1 year ago

myclamm commented 1 year ago

Clearly the answer is 240. But the data set says 150.

Question: Marin and his neighbor Nancy each eat 4 apples a day. How many apples do they eat in 30 days? Answer: 2 4 30 = 240

myclamm commented 1 year ago

@vineetsk10 can you pls accept PR, otherwise nothing will ever be able to get 100% on GSM8k.

vineetsk10 commented 1 year ago

Hi Mike, thanks for the PR. We however will not be merging any changes to the test set even if the questions are broken, in order to maintain consistency between previous reported and published eval results (otherwise historical comparisons would no longer be meaningful). As noted in Section 2 of the GSM8K paper, we estimate that less than 2 percent of problems contain breaking errors; that error rate should be taken into account when considering perfect performance or human-level performance.

Best, Vineet