Encode an empty string gives empty tokens

[flexwang]

tiktoken.encode('') will return an empty list. However, for huggingface tokenizer, encode an empty string will yield eos_token. Is there a way to make tiktoken behave the same way?

[pratyakshagarwal]

i added a contidion where if the text input is empty then it's gonna return the eos_token

[hauntsaninja]

Use tiktoken.encode('') or [eos_token]