AllYourBot / hostedgpt

An open version of ChatGPT you can host anywhere or run locally.
MIT License
344 stars 165 forks source link

Draft: Add token counting per message #460

Closed lumpidu closed 2 weeks ago

lumpidu commented 1 month ago

Implement token counting per message by using the returned usage stats from the LLM API backend. Anthropic sends token stats already by default. As for OpenAI, the stream needs to be first configured to return those.

This MR is related to #402

krschacht commented 2 weeks ago

I can't push some fixes to this branch and I want to get it merged in, so I needed to create a new PR: https://github.com/AllYourBot/hostedgpt/pull/495