Consistent Metric for OSS Model Cost

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

https://gorilla.cs.berkeley.edu/

Apache License 2.0

11.44k stars 992 forks source link

Consistent Metric for OSS Model Cost #533

Open HuanzhiMao opened 3 months ago

HuanzhiMao commented 3 months ago

We need to be consistent in our metrics to determine the cost for OSS models. If a model is hosted locally and has OSS_LATENCY, then it should not belong to the NO_COST_MODELS because we can use the following formula to estimate its cost.

ShishirPatil commented 1 month ago

Shall we close this?

HuanzhiMao commented 1 month ago

Shall we close this?

Not yet.