search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.79k stars 487 forks source link

Docs model ladder #708

Open IanMagnusson opened 3 months ago

IanMagnusson commented 3 months ago

Trying to add some documentation for the model ladder (per https://github.com/allenai/OLMo/issues/683) as I figure out how to use it.

IanMagnusson commented 3 months ago

Ideally we don't have to use --s3 for the usage example, but weka seems to be broken for a number of reasons right now.

  1. the current dolma data mix with the flan with rules based sep is not in weka
  2. If I revert to an earlier version of the ladder that didn't have this version of flan it still crashes without finding training data in weka although that data is there when I check manually.
IanMagnusson commented 3 months ago

Would be nice to also have the -h response for the other commands in the script (is there an easier way to get these than to do a gantry command to run this with the env all set up. When I try to run it locally I get env issues even tho I have OLMo pip instal -e

Likewise it would maybe be nice to list currently available sizes, lengths, and data mixes.