search

hendrycks / math

The MATH Dataset (NeurIPS 2021)
MIT License
879 stars 85 forks source link

math_equivalence.is_equiv behaves incorrectly when the model output is a floating-point number and the correct answer is an integer #15

Open coderfengyun opened 1 year ago

coderfengyun commented 1 year ago

As the title describes, math_equivalence.is_equiv fails to work properly when the model output is a floating point number and the correct answer in the dataset is an integer.

Provide a test case that should pass as expectation:

def test_decimals_integer(self):
    test_in = "1.0"
    test_out = "1"
    self.assertTrue(is_equiv(test_in, test_out))

I hope to communicate on this issue and if it is confirmed that it should be supported, I can come and provide the pull request.