EVAL-MGSM-Langbridge

[facemyself]

Thank you for this wonderful work. Is there a problem with the evaluation method? I used scripts/eval/mgsm/metamath-lb-9B.bash（checkpoint：kaist-ai/metamath-langbridge-9b and kaist-ai/langbridge_encoder_tokenizer）, but the eval result is this and very lower	Task	Version	Metric	Value		Stderr
mgsm_bn	0	acc	0.048	±	0.0135
mgsm_de	0	acc	0.104	±	0.0193
mgsm_en	0	acc	0.128	±	0.0212
mgsm_es	0	acc	0.096	±	0.0187
mgsm_fr	0	acc	0.100	±	0.0190
mgsm_ja	0	acc	0.052	±	0.0141
mgsm_ru	0	acc	0.052	±	0.0141
mgsm_sw	0	acc	0.028	±	0.0105
mgsm_te	0	acc	0.036	±	0.0118
mgsm_th	0	acc	0.072	±	0.0164
mgsm_zh	0	acc	0.048	±	0.0135

And if I want to train and evaluate using this method, should I add the save of lm.tokenizer and enc.tokenizer in train_langbridge?

[Kosei1227]

This score table is quite similar to the one I got previously. Could you try to use the transformer version specified in the requirement.txt? Hopefully, this will fix your issue.

[MattYoon]

Thank you for reporting, please refer to https://github.com/kaistAI/LangBridge/issues/11