search

pkoukk / tiktoken-go

go version of tiktoken
MIT License
601 stars 67 forks source link

feat: support o200k_base (gpt-4o) #47

Closed WqyJh closed 1 month ago

WqyJh commented 1 month ago
sash20m commented 1 month ago

Do you happen to know what is the token count for gpt-4o ? They said it's less then previous models, it it's probably less than 3 (as for gpt-4) Refference: https://github.com/pkoukk/tiktoken-go#counting-tokens-for-chat-api-calls

WqyJh commented 1 month ago

Yes it's for gpt-4o

sash20m commented 1 month ago

Yes it's for gpt-4o

Yes, but I mean what is the token per messge, as in the number. Is it 3 like for gpt-4 or 4 like for gpt-3.5 (unlikely). If you know of course, it's possible to calculate with code the link provided above but haven't tried.

WqyJh commented 1 month ago

Well, currently not sure. I didn't find any clusion of it currently.

But I believe it's the same with gpt-4 currently. I've just say hi to gpt-4o and gpt-3.5-turbo, the input/output tokens is the same. And I count the tokens when stream=true, it's also the same.

image

If openai didn't mention that it's changed, then it's probably not changed. But it might be changed after the voice & video capabilities of gpt-4o was released.