search

facebookresearch / MobileLLM

MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.
Other
1.16k stars 65 forks source link

Evaluation pipeline #14

Open kalyani7195 opened 1 month ago

kalyani7195 commented 1 month ago

Hi Zechun, great work! Could you please share the details about the evaluation code? like which codebase was used to run inference etc. thank you, Kalyani

galk2 commented 1 month ago

Would like to get my hand on the evaluation code. For example, do you prompt the model for evaluating the zero shot accuracy and do you take argmax over the whole vocab or only on A/B/C/D. Thanks!