Open kylewlacy opened 10 months ago
Good point @kylewlacy. Do you know if OpenAI has any documentation on how tokens are counted?
Their cookbook documentation at https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken doesn't seem to mention functions
I suspect part of the issue is that the "function_call" property is not considered when calculating the length of a message.
See the code below making use of role
and content
while function_call
is unused.
https://github.com/zurawiki/tiktoken-rs/blob/8b5dc5482a6a3e7a261d8f59238c70e3d04e5ecd/tiktoken-rs/src/api.rs#L117-L129
When using the relatively recent "Functions" feature of the ChatGPT API, it seems like
tiktoken-rs
underestimates the total number of tokens in the request. Here's a minimal example request:I get this response from OpenAI:
...indicating the request consumed 78 tokens for the prompt. However,
tiktoken_rs::num_tokens_from_messages
returns a value of 66 tokens.