Track model metadata - Githubissues

dimagi / open-chat-studio

A web based platform for building Chatbots backed by Large Language Models

BSD 3-Clause "New" or "Revised" License

13 stars 7 forks source link

Track model metadata #481

Open SmittieC opened 1 week ago

SmittieC commented 1 week ago

It can really help users out if OCS knows a few things about selected models so that we can build guardrails that will ultimately help lower frustration and make the platform for robust. See this thread as an example where it would have been useful if OCS had known what the model's token limit is.

Model metadata to track

Token limit
Rates / cost
Support for function calling

SmittieC commented 5 days ago

If we know the token limit for a specific model, we can

Do do input token count and disallow users to input messages larger than that which the model can handle (on webusers though)
Do proper limiting and/or estimation for what the max token limit should be. Currently users can set this to any number, regardless of the model's context limit.