Fixes some bugs to enable evaluation of models stored in subfolders. Changes are as follows:
Fixes inconsistency between model_name_or_path vs. model as a command line arg, and similarly for tokenizer_name_or_path vs. tokenizer; the _name_or_path version are used in the actual eval scripts, but in the yaml template and and at some places in the submit_eval_jobs script, model and tokenizer are used instead. I made it consistent to always use model_name_or_path, and tokenizer_name_or_path. Previously, some of the code attempted to change model to a subdirectory, which caused an error.
Makes it so that passing a subdirectory changes the model_name_or_path but not tokenizer_name_or_path. We want this because the training code by default doesn't dump a separate tokenizer for each subdirectory, it only saves one to the top level. The current behavior caused an error because it couldn't find the tokenizer.
Explicitly imports OlmoTokenizerFast in utils.py. Without this import, HF can't find the OLMo tokenizer and throws an error.
As far as I can tell things work now; I ran some evals and the code seems to properly find model subdirectories and is able to evaluate.
Fixes some bugs to enable evaluation of models stored in subfolders. Changes are as follows:
model_name_or_pathvs.modelas a command line arg, and similarly fortokenizer_name_or_pathvs.tokenizer; the_name_or_pathversion are used in the actual eval scripts, but in theyamltemplate and and at some places in thesubmit_eval_jobsscript,modelandtokenizerare used instead. I made it consistent to always usemodel_name_or_path, andtokenizer_name_or_path. Previously, some of the code attempted to changemodelto a subdirectory, which caused an error.model_name_or_pathbut nottokenizer_name_or_path. We want this because the training code by default doesn't dump a separate tokenizer for each subdirectory, it only saves one to the top level. The current behavior caused an error because it couldn't find the tokenizer.OlmoTokenizerFastinutils.py. Without this import, HF can't find the OLMo tokenizer and throws an error.As far as I can tell things work now; I ran some evals and the code seems to properly find model subdirectories and is able to evaluate.