search

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
https://gpt-tokenizer.dev
MIT License
434 stars 35 forks source link

Add gpt-4-32k to mappings #13

Closed wesbos closed 1 year ago

wesbos commented 1 year ago

Needed for the work on https://github.com/niieani/gpt-tokenizer-demo/pull/3

I dont have access to 32k but from looking at code examples, this is the name of the 32k GPT4 model

niieani commented 1 year ago

Thanks @wesbos. Added this along with the addition of encodeChat function.