OLMo-2 held-out validation data

allenai / OLMo

Modeling, training, eval, and inference code for OLMo

https://allenai.org/olmo

Apache License 2.0

4.79k stars 488 forks source link

OLMo-2 held-out validation data #755

Open chawins opened 3 days ago

chawins commented 3 days ago

❓ The question

I'm very excited about the release of OLMo-2, and thank you for doing all the hard work!

I noticed that the config file of OLMo-2 7B and 8B has commented out the validation set used in the first OLMo. Why is this the case? All those data (along with Paloma?) were no longer held out and used in the pre-training?