search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.31k stars 833 forks source link

remove duplicated code #79

Closed GLEF1X closed 1 year ago