search

lm-sys / RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!
Apache License 2.0
2.78k stars 204 forks source link

Question: was text-embedding-3-small included in the bills? #5

Closed Sunt-ing closed 1 month ago

Sunt-ing commented 1 month ago

Hi developers, thanks for your effort on this project!

I have a question: in the paper, when calculating router monetary costs, were the costs of the OpenAI embedding model (i.e., text-embedding-3-small) included?

iojw commented 1 month ago

Hi, are you referring to the cost savings section (4.4) in the paper? For this, we calculated cost savings based on the number of GPT-4 calls only, since this is much greater (>100x) as compared to the cost for Mixtral 8x7B and embedding calls.

GPT-4: ~$20 / M tokens Mixtral: ~$0.20 / M tokens text-embedding-3-small: $0.02 / M tokens.

Let me know if this helps.

Sunt-ing commented 1 month ago

Thanks for your calculation!