openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12k stars 818 forks source link

Add gpt-4-1106-preview model #214

Closed JosephTLyons closed 7 months ago