search

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
https://gpt-tokenizer.dev
MIT License
451 stars 38 forks source link

Picture tokens #41

Open arthurwolf opened 9 months ago

arthurwolf commented 9 months ago

Hello.

The new -vision model lets you pass images (which end up as tokens). To properly calculate token use for my stuff, I'd need to be able to pass my full conversation/payload, which includes images. Does this tokenizer support that (reading the docs I don't see signs it does). Would you have recommendations of how to implement that on my side? And if I do implement it would that be a welcome PR?

Cheers.

niieani commented 3 months ago

No support at this moment, but this would definitely be a welcome PR! Thanks Arthur.

henrymcl commented 3 months ago

Hello.

The new -vision model lets you pass images (which end up as tokens). To properly calculate token use for my stuff, I'd need to be able to pass my full conversation/payload, which includes images. Does this tokenizer support that (reading the docs I don't see signs it does). Would you have recommendations of how to implement that on my side? And if I do implement it would that be a welcome PR?

Cheers.

How will you be supplying your images? In base64 or external URL?

arthurwolf commented 3 months ago

Either is fine, I use base64 right now.