search

truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2.03k stars 176 forks source link

[BUG] Using AzureOpenAI provider does not show token usage/price #1152

Open Dannyjelll opened 3 months ago

Dannyjelll commented 3 months ago

Bug Description Cost/Token Usage display does not work with AzureOpenai Provider

To Reproduce Try to evaluate a RAG application using the AzureOpenai provider for any metric like groundedness or answer relevance

Expected behavior It should display the Token usage/price

Relevant Logs/Tracebacks I implemented both a custom retriever and one using Trullama. In both cases, I used AzureOpenAI as LLM.:

from trulens_eval.feedback.provider.openai import AzureOpenAI as fAzureOpenai openai_provider = fAzureOpenai(deployment_name="chat_test")

then f_qa_relevance = ( Feedback( openai_provider.relevance_with_cot_reasons, name = "Answer Relevance", ) .on(Select.RecordCalls.retrieve.args.query) .on_output() )

and so on. The usage is always 0.

Environment:

Additional context none

dosubot[bot] commented 3 months ago

The issue with the cost/token usage display not working for the AzureOpenAI provider when evaluating a RAG application for metrics like groundedness or answer relevance is likely due to incomplete or incorrect cost tracking implementation for Azure OpenAI models.

Steps to Resolve:

  1. Verify Environment Variables: Ensure that the following environment variables are correctly set:

    • OPENAI_API_VERSION
    • OPENAI_API_TYPE
    • AZURE_OPENAI_DEPLOYMENT_NAME
    • AZURE_OPENAI_ENDPOINT

  2. Check Cost Tracking Implementation: Review the AzureOpenAI class to ensure that cost tracking is correctly implemented. Verify that the cost tracking logic is integrated and that API responses are parsed to extract cost information.

  3. Debugging: Add logging or debugging statements in the AzureOpenAI class to trace the cost tracking logic and identify any failures.

  4. Update Trulens: Make sure you are using the latest version of Trulens, as updates or bug fixes might address this issue.

If the problem persists, further investigation into the AzureOpenAI class and the _test_llm_provider_endpoint method may be necessary to identify any missing implementations related to cost tracking.

#### Details **Sources** - [trulens_eval/tests/e2e/test_endpoints.py](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_endpoints.py) - [trulens_eval/examples/expositional/models/azure_openai_llama_index.ipynb](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/azure_openai_llama_index.ipynb)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

sfc-gh-jreini commented 2 months ago

@sfc-gh-dhuang can you take a look at this?