search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.76k stars 801 forks source link

- #130

Closed jsnijdewint closed 1 year ago

jsnijdewint commented 1 year ago

-