WARNING:root:Warning: model not found. Using cl100k_base encoding.

AgentOps-AI / tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

https://agentops.ai

MIT License

1.43k stars 53 forks source link

WARNING:root:Warning: model not found. Using cl100k_base encoding. #38

Open moresearch opened 5 months ago

moresearch commented 5 months ago

I am getting WARNING:root:Warning: model not found. Using cl100k_base encoding. when using non openAI models such as claude-2. I checked the code and it seems to be a tiktoken issue.

Am I getting wrong cost??

areibman commented 5 months ago

Potentially, yes. I believe Claude 1 uses cl100k, but Claude 2 and Claude 3 I'm not so sure.

Here is the only info I've found on Claude 3: https://github.com/javirandor/anthropic-tokenizer

And other Anthropic models: https://github.com/anthropics/anthropic-sdk-python/blob/bd866a1e837febd05b6f5cea322f1c42e29f2d01/src/anthropic/tokenizer.json

For stuff like this, TokenCost will likely be a close but not exact price estimate.