allenai / open-instruct

Apache License 2.0
1.08k stars 140 forks source link

Tokenizer path fix #137

Closed dwadden closed 3 months ago

dwadden commented 3 months ago

Previously, model_name_or_path was replaced twice, rather than replacing model_name_or_path.

I also added an olmo flag so that the code will do the right thing for OLMo models in cases where "olmo" isn't in the directory name.