In many cases it is useful to keep track of the token usage for each query and associated costs. Different model provider apis handle this differently (some provide this information as part of a generate query response) and the task is to return accurate usage states in a unified format for all model providers
Work Items
[x] OpenAI/AzureOpenAI
[x] HuggingFace
[ ] PALM
[ ] Extend response from palm api to include usage information.
What
In many cases it is useful to keep track of the token usage for each query and associated costs. Different model provider apis handle this differently (some provide this information as part of a generate query response) and the task is to return accurate usage states in a unified format for all model providers
Work Items
Palm api response provides the following fields.