[BUG] Trulens does not evaluate the cost per inference in the case of Bedrock.

[DhruvinKoshiya]

Bug Description I'm trying to evaluate RAG using Trulense with the Bedrock Llama 2 model, but in the Trulense dashboard, it does not provide the cost of inference. It only shows $0. If I use OpenAI instead of Bedrock Llama 2, then it shows the price for every inference.

To Reproduce

llm = Bedrock( model="meta.llama2-70b-chat-v1", profile_name=profile_name)

tru_recorder = TruChain(chain,app_id='ChatApplication', feedbacks=[feedback function])

with tru_recorder as recording:
    llm_response = llm.chat("What's the capital of the USA?")
display(llm_response)

Expected behavior Trulens should provide the cost for every inference in any LLM.

Relevant Logs/Tracebacks Trulens dashboard using Bedrock Llama 2 model.

Environment:

• OS: Windows
• Python Version: 3.10.1
• TruLens version: 0.27.0
• Versions of other relevant installed libraries: llama-index==0.10.25

[dosubot[bot]]

It seems that Trulens does not support cost tracking for Bedrock models, including the Bedrock Llama 2 model you're using. This limitation is noted in the Trulens codebase, specifically in tests for Bedrock models where cost tracking is explicitly disabled. This is because Trulens currently only supports cost tracking for OpenAI models.

For Bedrock models, while there is an indication that cost tracking might be possible through the Litellm integration, it's mentioned that this could be inaccurate. This suggests that any cost tracking for Bedrock models would need to be implemented manually and might not align perfectly with Trulens' existing cost tracking features.

If cost tracking is critical for your use case with Bedrock models, you might need to look into custom solutions or workarounds that can approximate the cost based on your specific usage patterns and the pricing model provided by Bedrock.

Sources

• trulens_eval/tests/e2e/test_endpoints.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[piotrm0]

Hi @DhruvinKoshiya , do you know if Bedrock models are charged at some fixed rate that can be determined from model name?

[DhruvinKoshiya]

@piotrm0 - Bedrock modes are charged per the input and output token. This document provides pricing of all the models.