search

AgentOps-AI / tokencost

Easy token price estimates for 400+ LLMs. TokenOps.
https://agentops.ai
MIT License
1.25k stars 41 forks source link

Fix finetuned gpt3.5 price (Feb 2024) #27

Closed oulianov closed 5 months ago

oulianov commented 5 months ago

Referring to the OpenAI pricing, there is an error in the LiteLLM prices file for finetuned models of gpt3.5

Capture d’écran 2024-02-04 à 17 59 32

The counting function would also fail with finetuned models.

krrishdholakia commented 5 months ago

fix pushed! thanks @oulianov @areibman

oulianov commented 5 months ago

Thank you for updating LiteLLM file, however this fix won't work until you strip the model name

krrishdholakia commented 5 months ago

@oulianov @areibman curious, what's missing in litellm.token_counter and litellm. cost_per_token to be helpful here?

https://docs.litellm.ai/docs/completion/token_usage#3-token_counter

areibman commented 5 months ago

@oulianov @areibman curious, what's missing in litellm.token_counter and litellm. cost_per_token to be helpful here?

https://docs.litellm.ai/docs/completion/token_usage#3-token_counter

Wish I'd known about this before! Few things I ran into that might be worth updating (if it's a priority for you):

  1. The way you calculate cost runs the risks of floating point precision errors. Either denominate everything as an int or use Decimal
  2. Prompts aren't always dictionaries. i.e. embedding models take strings
oulianov commented 5 months ago

I'm more concerned about long term maintainability of this pricing file. Obviously, pricing will keep changing. Providers will keep appearing. Rules will keep changing. There may be the need to price token cost in novel, advanced way (eg: using underlying GPU infracost ; automatically scrapping the price for websites) Having a dedicated repo and community caring about only this topic is a good thing imho 👍