search

kaistAI / LangBridge

[ACL 2024] LangBridge: Multilingual Reasoning Without Multilingual Supervision
https://aclanthology.org/2024.acl-long.405/
81 stars 7 forks source link

EVAL-MGSM-Langbridge #14

Closed zwRuan closed 1 month ago

zwRuan commented 2 months ago
Thank you for this wonderful work. Is there a problem with the evaluation method? I used scripts/eval/mgsm/metamath-lb-9B.bash（checkpoint：kaist-ai/metamath-langbridge-9b and kaist-ai/langbridge_encoder_tokenizer）, but the eval result is this and very lower Task Version Metric Value Stderr
mgsm_bn 0 acc 0.048 ± 0.0135
mgsm_de 0 acc 0.104 ± 0.0193
mgsm_en 0 acc 0.128 ± 0.0212
mgsm_es 0 acc 0.096 ± 0.0187
mgsm_fr 0 acc 0.100 ± 0.0190
mgsm_ja 0 acc 0.052 ± 0.0141
mgsm_ru 0 acc 0.052 ± 0.0141
mgsm_sw 0 acc 0.028 ± 0.0105
mgsm_te 0 acc 0.036 ± 0.0118
mgsm_th 0 acc 0.072 ± 0.0164
mgsm_zh 0 acc 0.048 ± 0.0135

And if I want to train and evaluate using this method, should I add the save of lm.tokenizer and enc.tokenizer in train_langbridge?

Kosei1227 commented 2 months ago

This score table is quite similar to the one I got previously. Could you try to use the transformer version specified in the requirement.txt? Hopefully, this will fix your issue.

MattYoon commented 2 months ago

Thank you for reporting, please refer to https://github.com/kaistAI/LangBridge/issues/11