search

epfl-dlab / transformers-CFG

🤗 A specialized library for integrating context-free grammars (CFG) in EBNF with the Hugging Face Transformers
http://saibo-creator.xyz:7860/
MIT License
90 stars 16 forks source link

Feature/add support for qwen2 models #97

Closed nathanrchn closed 2 months ago

nathanrchn commented 2 months ago

This PR introduces support for the Qwen2TokenizerFast tokenizer, enabling the use of all Qwen2 and Qwen2.5 models.

I found that Qwen2TokenizerFast is the same tokenizer as the GPT2TokenizerFast. Therefore, I routed the Qwen2 tokenizer to the GPT2 tokenizer for compatibility source.