Avoid calling byte_pair_encode for existing tokens

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

12.52k stars 856 forks source link

Closed hauntsaninja closed 9 months ago

hauntsaninja commented 9 months ago

This was byte_pair_encode can be optimized further, assuming we'll always have at least 2 tokens

Authored by @paplorinc in #239