Add better token counting

[crspeller]

Token counting is a bit hard to get right. The current implmentation for OpenAI is an approximation, see: https://github.com/mattermost/mattermost-plugin-ai/blob/master/server/ai/openai/openai.go#L296. Others have worse.

Implement better counting (or figure out another solution) for:

• [ ] OpenAI
• [ ] Anthropic
• [ ] AskSage

[azigler]

Are there resources about token counting that someone could read to start learning about this topic? What context can we provide so the difficulty is understood?

[ishaan-jaff]

Hi @azigler, @crspeller I'm the maintainer of LiteLLM https://github.com/BerriAI/litellm we allow you to do cost tracking for 100+ LLMs

Usage

Docs: https://docs.litellm.ai/docs/#calculate-costs-usage-latency

from litellm import completion, completion_cost
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="gpt-3.5-turbo", 
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

cost = completion_cost(completion_response=response)
print("Cost for completion call with gpt-3.5-turbo: ", f"${float(cost):.10f}")

We also allow you to create a self hosted OpenAI Compatible proxy server to make your LLM calls (100+ LLMs), track costs, token usage Docs: https://docs.litellm.ai/docs/simple_proxy

I hope this is helpful, if not I'd love your feedback on what we can improve

[crspeller]

This isn't really about cost it's about token counting for fitting text into the LLM context window. We also can't rely on something like LiteLLM being in the middle for all deployments.

[azigler]

Hey, Mattermost AI community! :wave:

We're looking for help on this one. :eyes: In this issue, we want to improve the plugin's token counting capabilities and accuracy for common LLM providers. Understanding token counting is vital for communicating with LLMs programmatically. Check out this help topic from OpenAI to learn more.

For those interested in tackling this, here are some starting points:

• Review the current implementation for OpenAI's token counting using this reference and improve the existing method.
• Evaluate and account for the other token limits for other LLM providers like Anthropic and Ask Sage.

Dive in and let us know if you need any guidance along the way. :thumbsup:

Happy contributing! :sparkles:

[darkLord19]

How about using some tokenizer library like tiktoken-go?