Closed FergusFettes closed 3 months ago
I got an error cause a file I was trying to count the tokens of contained some special tokens. This is a quick fix for that. Happy to make some modifications or write a test if you are interested in this change.
Before:
echo '<|endoftext|>' | ttok > Long tiktoken error
Now:
echo '<|endoftext|>' | ttok --allowed-special='<|endoftext|>' 2
Thanks for the PR! I ended up solving this a slightly different way:
I got an error cause a file I was trying to count the tokens of contained some special tokens. This is a quick fix for that. Happy to make some modifications or write a test if you are interested in this change.
Before:
Now: