Open pamelafox opened 5 months ago
It will be great to have a cost calculator or at least a token calculator within tiktoken for prompt as well as for messages in chatCompletion
I agree, it seems like it's constantly changing.
@stephenasuncionDEV I'm curious, have you seen a change in the logic needed for the calculation I pu above? Just want to make sure I didn't miss an announcement.
For now, since I have a need to use this functionality across multiple projects, I've put it in a small package: https://github.com/pamelafox/openai-messages-token-helper
For example-
from openai_messages_token_helper import count_tokens_for_image
image = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEA..."
num_tokens = count_tokens_for_image(image)
Will happily move to tiktoken or openAI if the functionality gets moved to one of those packages, though.
We have implemented a lot of logic around token counting for ChatCompletion requests, and it feels like the logic should go in a separate package. I'm wondering if tiktoken would be an appropriate spot, given the logic all depends on tiktoken?
Specifically, I'm thinking of this sort of code, which is based off cookbooks:
We also have full tests for that code.
Would that be appropriate for tiktoken, or is it already in a separate package? It seems like it'd be helpful to be packaged up for easier community re-use. Thanks!