Closed myclamm closed 1 year ago
@vineetsk10 can you pls accept PR, otherwise nothing will ever be able to get 100% on GSM8k.
Hi Mike, thanks for the PR. We however will not be merging any changes to the test set even if the questions are broken, in order to maintain consistency between previous reported and published eval results (otherwise historical comparisons would no longer be meaningful). As noted in Section 2 of the GSM8K paper, we estimate that less than 2 percent of problems contain breaking errors; that error rate should be taken into account when considering perfect performance or human-level performance.
Best, Vineet
Clearly the answer is 240. But the data set says 150.
Question: Marin and his neighbor Nancy each eat 4 apples a day. How many apples do they eat in 30 days? Answer: 2 4 30 = 240