search

mvitlov / tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models
Other
23 stars 7 forks source link

Refactor to reduce compile size (via tree-shaking) #4

Closed mvitlov closed 1 year ago