search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.16k stars 751 forks source link

Store tokens in u32 instead of usize #251

Closed hauntsaninja closed 5 months ago

hauntsaninja commented 5 months ago

And hide it behind a Rank type to make it easier to separate it from other numeric values

Authored by @paplorinc in https://github.com/openai/tiktoken/pull/239