One of the biggest issues with GSM-8K or any other dataset openly available on the internet, is that larger models inherently have the solutions as part of the training set when they crawl the entire internet.
One way to change would be to have this repo be able to dynamically generate a test set where the numbers, names and formatting are different everytime a new set is generated. The deeper semantic logic would still be the same, but it would force the model not to memorize even if the raw version of test dataset is openly available on internet for crawling.
One of the biggest issues with GSM-8K or any other dataset openly available on the internet, is that larger models inherently have the solutions as part of the training set when they crawl the entire internet.
https://twitter.com/suchenzang/status/1701615029211238904
https://arxiv.org/abs/2309.08632 (pre-training on the dataset is all you need).
One way to change would be to have this repo be able to dynamically generate a test set where the numbers, names and formatting are different everytime a new set is generated. The deeper semantic logic would still be the same, but it would force the model not to memorize even if the raw version of test dataset is openly available on internet for crawling.