search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k stars 856 forks source link

Inline custom mapping function in _byte_pair_merge #253

Closed hauntsaninja closed 9 months ago

hauntsaninja commented 9 months ago

Authored by @paplorinc in #239