Closed grantjenks closed 3 months ago
I just ran cross this myself: https://twitter.com/simonw/status/1786172033245597997
ttok '<|endoftext|>' --encode
I'm going to add a --allow-special
option.
Filed this related issue:
ttok '<|endoftext|>' --encode --allow-special
100257
Would be nice to have options for special token handling. I got this error today: