Closed kangzhiq closed 2 months ago
Sorry, I just got the idea of operator induction: to infer the operator from the context and calculate the correct output.
Closing this issue.
Sorry, I just got the idea of operator induction: to infer the operator from the context and calculate the correct output.
Closing this issue.
Exactly, that's right.
Thanks for the great work!
I just noticed that the evaluation protocol of the Operator Induction dataset is not consistent with what has been presented in the paper. The dataset was designed to induce the operator. However, in your implementation, the model is asked to guess
What is the result of the following mathematical expression?
and the evaluation is also based on the output digit.Could you please elaborate more on this? For me, it does not make sense to ask the model to guess the output of 2 ?3 and say the ground-truth is 6 instead of 5
Thanks in advance!