search

truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2k stars 170 forks source link

[BUG] Azure OpenAI error with evaluation using CustomApp #1284

Open dtkettler opened 1 month ago

dtkettler commented 1 month ago

Bug Description

When looking for information on this I found another issue that this seems similar to: https://github.com/truera/trulens/issues/1071 (which was never really resolved?). Though in my case I am using TruCustomApp rather than Llamaindex.

The issue is that when running with a recorder and TruCustomApp I get these warnings: WARNING:trulens_eval.feedback.provider.endpoint.base:Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. and then eventually the error: openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

However before you say it's an issue with my provider, I've actually run some tests with invoking the provider directly and that works fine. That is to say I can run things like provider.relevance(prompt=question, response=response) where I manually insert the appropriate text and there are zero issues. But when I use the same provider (not just same settings, literally the same instance) inside of an app with recording suddenly it claims there's no endpoint.

To Reproduce Below is a sample of my code, and yes I know it's kind of odd. It's using LangChain but I could note utilize TruChain because I am working with an existing code base that was written in a bit of a non-standard way. doc_search here is a LangChain tool and it's essentially functioning as a retriever but it doesn't actually inherent from BaseRetriever. And as such I could not figure out how to get context when using TruChain. That's the entire reason I went with TruCustomApp so I could specify it manually. Though I then call retrieve_chunks from respond_to_query even though the result isn't used there just to try and log things. That means it actually gets called twice each iteration because it's used again by agent_chain. It doesn't seem like a great way of handling things but I'm not sure what a better way would be without rewriting the code I'm trying to evaluate here.

Anyway, I don't see how any of that would be related to the provider errors I'm getting.

provider = AzureOpenAI(
    deployment_name=MODEL,
    api_key=get_apim_key(MODEL),
    azure_endpoint=os.environ["APIM_ENDPOINT"] + MODEL,
    api_version=os.environ["AZURE_OPENAI_API_VERSION"]
)

class CustomApp:
    def __init__(self, doc_search, agent_chain):
        self.doc_search = doc_search
        self.agent_chain = agent_chain

    @instrument
    def retrieve_chunks(self, query):
        context = self.doc_search.run(query)
        return [x.page_content for x in context]

    @instrument
    def respond_to_query(self, input):
        context = self.retrieve_chunks(input)

        response = self.agent_chain.invoke({"input": input})
        return response['output']

f_groundedness = Feedback(provider.groundedness_measure_with_cot_reasons).on(Select.RecordCalls.retrieve_chunks.rets).on(Select.RecordCalls.respond_to_query.rets)
f_answer_relevance = Feedback(provider.relevance).on(Select.RecordCalls.respond_to_query.args.input).on(Select.RecordCalls.respond_to_query.rets)
f_context_relevance = Feedback(provider.context_relevance_with_cot_reasons).on(Select.RecordCalls.retrieve_chunks.args.query).on(Select.RecordCalls.retrieve_chunks.rets.collect()).aggregate(np.mean)

ca = CustomApp(doc_search, agent_chain)
tru_recorder = TruCustomApp(ca, app_id="Test", feedbacks=[f_context_relevance, f_groundedness, f_answer_relevance])

with open("test-queries.csv") as csvfile:
    reader = csv.reader(csvfile)

    with tru_recorder as recording:
        for num, row in enumerate(reader):
            question = row[0]

            ca.respond_to_query(question)

Expected behavior I would expect to run the app with recording without errors.

Relevant Logs/Tracebacks

WARNING:trulens_eval.feedback.provider.endpoint.base:Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
WARNING:trulens_eval.feedback.provider.endpoint.base:Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
WARNING:trulens_eval.feedback.provider.endpoint.base:Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
ERROR:trulens_eval.instruments:Error calling wrapped function respond_to_query.
ERROR:trulens_eval.instruments:Traceback (most recent call last):
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\trulens_eval\instruments.py", line 511, in tru_wrapper
    rets, cost = mod_endpoint.Endpoint.track_all_costs_tally(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\trulens_eval\feedback\provider\endpoint\base.py", line 496, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\trulens_eval\feedback\provider\endpoint\base.py", line 477, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\trulens_eval\feedback\provider\endpoint\base.py", line 574, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DKettle1\AppData\Local\Temp\ipykernel_23040\2912314505.py", line 16, in respond_to_query
    response = self.agent_chain.invoke({"input": input})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\langchain\chains\base.py", line 166, in invoke
    raise e
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\langchain\chains\base.py", line 156, in invoke
    self._call(inputs, run_manager=run_manager)
  File "c:\Users\DKettle1\code\functionapp\azure-func-qtm-kb-search\.venv\Lib\site-packages\langchain\agents\agent.py", line 1433, in _call
...
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

Environment:

dosubot[bot] commented 1 month ago

Hi @dtkettler! I'm here to help with any bugs, questions, or contributions you might have. Let's see what we can do to solve the issue you're facing!

To resolve the Azure OpenAI endpoint not found error when running a recorder with TruCustomApp and an Azure OpenAI provider, ensure that the Azure OpenAI endpoint is correctly configured. Here is an example setup:

from trulens_eval.keys import check_or_set_keys

# Set your Azure OpenAI endpoint and key
check_or_set_keys(
    AZURE_OPENAI_ENDPOINT="your_azure_openai_endpoint",
    AZURE_OPENAI_API_KEY="your_azure_openai_api_key",
    OPENAI_API_VERSION="your_openai_api_version"
)

# Initialize the Azure OpenAI provider
from trulens_eval.feedback.provider.openai import AzureOpenAI
azure_openai = AzureOpenAI(deployment_name="your_deployment_name")

# Use the Azure OpenAI provider in your application
response = azure_openai.relevance(
    prompt="Where is Germany?",
    response="Poland is in Europe."
)
print(response)  # This should print the relevance score

Make sure to replace "your_azure_openai_endpoint", "your_azure_openai_api_key", "your_openai_api_version", and "your_deployment_name" with your actual Azure OpenAI endpoint, API key, API version, and deployment name.

If the error persists, double-check that the endpoint URL, API key, and deployment name are correct and that your network configuration allows access to the Azure OpenAI service [1][2].

To continue talking to Dosu, mention @dosu.