search

marella / ctransformers

Python bindings for the Transformer models implemented in C/C++ using GGML library.
MIT License
1.82k stars 138 forks source link

Llama tokenizer can not stop at </s> #124

Open lucasjinreal opened 1 year ago

lucasjinreal commented 1 year ago 
Sure, why not? What would you like to know?</s>

why it can not stop at stop tokenizer, it will print out this token