search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.56k stars 273 forks source link

dynamic enable special tokens #164

Closed wangyu1997 closed 8 months ago

wangyu1997 commented 11 months ago

remove "encode/decode_special_tokens" parameters in encode/decode method, add global enable_special_tokens option and turn on depend on the existence of add_tokens.json

turboderp commented 11 months ago

This would be a breaking change to the API. And quite a few use-cases rely on controlling when added tokens are encoded or decoded. E.g. not encoding special tokens removes the need to sanitize user input, and it's necessary if you want to be able to encode something like </s> as text rather than as a control symbol.