How to get the MMLU results in Table 4?

allenai / OLMoE

OLMoE: Open Mixture-of-Experts Language Models

https://arxiv.org/abs/2409.02060

Apache License 2.0

478 stars 37 forks source link

Open mathfinder opened 2 months ago

mathfinder commented 2 months ago

Thank you so much for your excellent work.

We find there are three types of MMLU results in wandb: mmlu mc 5shot mmlu mc 5 shot test mmlu var

Which one you use in the table 4?

Muennighoff commented 2 months ago

Thanks! Those are not in WandB but run via OLMES as mentioned in the table caption. See https://github.com/allenai/OLMoE?tab=readme-ov-file#after-pretraining for running that :)