Closed apaniukov closed 4 months ago
not-add-special-tokens
do we need to merge it to release branches too?
We could port the first commit with the fix for llama3 tokenizer to 2024.1.0.1 release.
not-add-special-tokens
flag during conversion