The calculation results different with Tokenizer

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.76k stars 801 forks source link

Closed mysaber closed 1 year ago

mysaber commented 1 year ago

Why are the calculation results different with Tokenizer

hauntsaninja commented 1 year ago

Depends on what encoding you use.

labdmitriy commented 1 year ago

Hello @hauntsaninja,

Could you please explain in more details? I've noticed also this behavior, how to use tiktoken to reproduce exactly the same result as Tokenizer?

Thank you.

eastack commented 1 year ago

How to count tokens with tiktoken at "6. Counting tokens for chat API calls"

labdmitriy commented 1 year ago

@eastack Thank you! I am familiar with it, I thought maybe it is something else 🙏