Open isamu-isozaki opened 4 months ago
This is a PR to let hugging face tokenizer handle prompt formatting and remove the need for the format_prompt scripts which we had to make for each model. I mainly tested on the phi3b model and it seemed to work
This is a PR to let hugging face tokenizer handle prompt formatting and remove the need for the format_prompt scripts which we had to make for each model. I mainly tested on the phi3b model and it seemed to work