mvitlov / tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models
Other
23 stars 7 forks source link