search

LYH-YF / MWPToolkit

MWPToolkit is an open-source framework for math word problem(MWP) solvers.
MIT License
162 stars 37 forks source link

Experiments for MAWPS-s #11

Open allanj opened 3 years ago

allanj commented 3 years ago

Is the experiment for MAWPS-s using 5-fold as well? It seems yes to me as the paper reported. I got around 85.4 accuracy on MAWPS using train/dev/test. Wondering if I'm correct here. image

LYH-YF commented 3 years ago

yes, mawps-s is 5-fold setting.

allanj commented 3 years ago

Thanks. Am I right that, for SVAMP, you are just directly doing train and test following the SVAMP paper?

LYH-YF commented 3 years ago

SVAMP is just a dataset for test, according to SVAMP paper, trainset consists of mawps and asdiv-a. And the setting is train-test split.running it with k-fold cross validation may not a good idea.

allanj commented 3 years ago

Got it. Maybe should specify them in the table/paper?

From the table, it seems only those marked with "*" are train-test split.

allanj commented 3 years ago

In the SVAMP paper, the appendix A show that the transformers with Roberta encoder obtain 38.9 accuracy image

But it seems the RobertaGen only get 30.3 here. Curious about the difference here