SEMOSS / Semoss

Apache License 2.0
28 stars 5 forks source link

Fix tokenization for Llama models with the HF tokenizer #217

Closed ryanweiler92 closed 2 weeks ago

ryanweiler92 commented 3 weeks ago

Adding unique format_with_chat_template() methods to the OpenAiTokenizer and HuggingfaceTokenizer classes.