Different calculating result between gpt-tokenizer and openai

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.

https://gpt-tokenizer.dev

MIT License

434 stars 35 forks source link

Closed Penggeor closed 1 year ago

Penggeor commented 1 year ago

I am confuse why the calculating result is different from openai.

kyeshmz commented 1 year ago

Same as #20