allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.37k stars 431 forks source link

Use olmo_data tokenizer in Tokenizer.from_train_config #647

Closed 2015aroras closed 1 month ago

2015aroras commented 1 month ago

Fixes a breaking change for training