Avoid calling byte_pair_encode for existing tokens

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

12k stars 818 forks source link

Closed hauntsaninja closed 7 months ago

hauntsaninja commented 7 months ago

This was byte_pair_encode can be optimized further, assuming we'll always have at least 2 tokens

Authored by @paplorinc in #239