Not using role for token calculation

[jayendranarumugam]

while the token calculation currently its only used to take tokens inside content alone, but actually gpt uses the role also when doing the calculation.

e.g,

    "messages": [
        {
            "role": "user",
            "content": "Hi"
        }
    ]
}

Is there any specific reason that it got not taken into consideration during the calculation part ?

[azureholic]

interesting find. Do you have a reference to this (that it is actually using the "message" and not the "content")?

[jayendranarumugam]

Not directly but what i could find from my testing is for every chat completion model using gpt3.5-turbo, gpt4-turbo, gpt4, gpt4o model, input prompt is getting adding with additional 7 tokens

for e.g,

{
    "messages": [
        {
            "role": "user",
            "content": "Hi"
        }
    ],
    "stream": true
}

This will not give token as 1 which is currently happening in the source code

But the actual promptoken calculated by gpt model is 8 and not 1

This is because it appends some reserved keywords that comes additional 7 tokens

<|im_start|>user
Hi<|im_end|>
<|im_start|>assistant

Try the above https://tiktokenizer.vercel.app/?model=gpt-4

---

Some reference:

https://community.openai.com/t/what-is-the-reason-for-adding-total-7-tokens/337002/8 https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb

[azureholic]

Thanks! I see that they have released token count for streaming as well in the API a month ago.

That would eliminate the need for custom token calculation.

I will look into this and see if this is available on Azure OpenAI as well.

https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156

[jayendranarumugam]

see if this is available on Azure OpenAI as well.

I think it’s not yet available on azure side.

---

Update

It is available from starting from 2024-09-01-preview version