Open Dannyjelll opened 3 months ago
The issue with the cost/token usage display not working for the AzureOpenAI provider when evaluating a RAG application for metrics like groundedness or answer relevance is likely due to incomplete or incorrect cost tracking implementation for Azure OpenAI models.
Verify Environment Variables: Ensure that the following environment variables are correctly set:
OPENAI_API_VERSION
OPENAI_API_TYPE
AZURE_OPENAI_DEPLOYMENT_NAME
AZURE_OPENAI_ENDPOINT
Check Cost Tracking Implementation: Review the AzureOpenAI
class to ensure that cost tracking is correctly implemented. Verify that the cost tracking logic is integrated and that API responses are parsed to extract cost information.
Debugging: Add logging or debugging statements in the AzureOpenAI
class to trace the cost tracking logic and identify any failures.
Update Trulens: Make sure you are using the latest version of Trulens, as updates or bug fixes might address this issue.
If the problem persists, further investigation into the AzureOpenAI
class and the _test_llm_provider_endpoint
method may be necessary to identify any missing implementations related to cost tracking.
To continue the conversation, mention @dosu.
@sfc-gh-dhuang can you take a look at this?
Bug Description Cost/Token Usage display does not work with AzureOpenai Provider
To Reproduce Try to evaluate a RAG application using the AzureOpenai provider for any metric like groundedness or answer relevance
Expected behavior It should display the Token usage/price
Relevant Logs/Tracebacks I implemented both a custom retriever and one using Trullama. In both cases, I used AzureOpenAI as LLM.:
from trulens_eval.feedback.provider.openai import AzureOpenAI as fAzureOpenai openai_provider = fAzureOpenai(deployment_name="chat_test")
then f_qa_relevance = ( Feedback( openai_provider.relevance_with_cot_reasons, name = "Answer Relevance", ) .on(Select.RecordCalls.retrieve.args.query) .on_output() )
and so on. The usage is always 0.
Environment:
Additional context none