I wanted to do something special on this exception in my own code, so I had to write this:
try:
tokens = encoding.encode(text, **kwargs)
except ValueError as ex:
if 'disallowed special token' in str(ex):
# Do something special
I suggest having a custom exception class for this instead:
class DisallowedSpecialTokenError(ValueError):
pass
Raising that class instead would let people like me catch it explicitly, and since it's a subclass of ValueError it should not break any existing code that currently catches ValueError directly.
This code here: https://github.com/openai/tiktoken/blob/39f29cecdb6fc38d9a3434e5dd15e4de58cf3c80/tiktoken/core.py#L375-L383
I wanted to do something special on this exception in my own code, so I had to write this:
I suggest having a custom exception class for this instead:
Raising that class instead would let people like me catch it explicitly, and since it's a subclass of
ValueErrorit should not break any existing code that currently catchesValueErrordirectly.