Options for special token handling

[grantjenks]

Would be nice to have options for special token handling. I got this error today:

ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

[simonw]

I just ran cross this myself: https://twitter.com/simonw/status/1786172033245597997

ttok '<|endoftext|>' --encode

I'm going to add a --allow-special option.

[simonw]

Filed this related issue:

• https://github.com/openai/tiktoken/issues/290

[simonw]

ttok '<|endoftext|>' --encode --allow-special

100257