Track Usage Metrics in Chat API. Track Running Cost, Accuracy in Evals

khoj-ai / khoj

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (e.g gpt, claude, gemini, llama, qwen, mistral).

https://khoj.dev

GNU Affero General Public License v3.0

16.19k stars 795 forks source link

Track Usage Metrics in Chat API. Track Running Cost, Accuracy in Evals #985

Closed debanjum closed 5 days ago

debanjum commented 6 days ago

Track, return cost and usage metrics in chat api response Track input, output token usage and cost of interactions with openai, anthropic and google chat models for each call to the khoj chat api
Collect, display and store costs & accuracy of eval run currently in progress This provides more insight into eval runs during execution instead of having to wait until the eval run completes.