Open dtkettler opened 1 month ago
Hi @dtkettler! I'm here to help with any bugs, questions, or contributions you might have. Let's see what we can do to solve the issue you're facing!
To resolve the Azure OpenAI endpoint not found error when running a recorder with TruCustomApp
and an Azure OpenAI provider, ensure that the Azure OpenAI endpoint is correctly configured. Here is an example setup:
from trulens_eval.keys import check_or_set_keys
# Set your Azure OpenAI endpoint and key
check_or_set_keys(
AZURE_OPENAI_ENDPOINT="your_azure_openai_endpoint",
AZURE_OPENAI_API_KEY="your_azure_openai_api_key",
OPENAI_API_VERSION="your_openai_api_version"
)
# Initialize the Azure OpenAI provider
from trulens_eval.feedback.provider.openai import AzureOpenAI
azure_openai = AzureOpenAI(deployment_name="your_deployment_name")
# Use the Azure OpenAI provider in your application
response = azure_openai.relevance(
prompt="Where is Germany?",
response="Poland is in Europe."
)
print(response) # This should print the relevance score
Make sure to replace "your_azure_openai_endpoint"
, "your_azure_openai_api_key"
, "your_openai_api_version"
, and "your_deployment_name"
with your actual Azure OpenAI endpoint, API key, API version, and deployment name.
If the error persists, double-check that the endpoint URL, API key, and deployment name are correct and that your network configuration allows access to the Azure OpenAI service [1][2].
To continue talking to Dosu, mention @dosu.
Bug Description
When looking for information on this I found another issue that this seems similar to: https://github.com/truera/trulens/issues/1071 (which was never really resolved?). Though in my case I am using TruCustomApp rather than Llamaindex.
The issue is that when running with a recorder and TruCustomApp I get these warnings:
WARNING:trulens_eval.feedback.provider.endpoint.base:Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.
and then eventually the error:openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}
However before you say it's an issue with my provider, I've actually run some tests with invoking the provider directly and that works fine. That is to say I can run things like
provider.relevance(prompt=question, response=response)
where I manually insert the appropriate text and there are zero issues. But when I use the same provider (not just same settings, literally the same instance) inside of an app with recording suddenly it claims there's no endpoint.To Reproduce Below is a sample of my code, and yes I know it's kind of odd. It's using LangChain but I could note utilize TruChain because I am working with an existing code base that was written in a bit of a non-standard way. doc_search here is a LangChain tool and it's essentially functioning as a retriever but it doesn't actually inherent from BaseRetriever. And as such I could not figure out how to get context when using TruChain. That's the entire reason I went with TruCustomApp so I could specify it manually. Though I then call retrieve_chunks from respond_to_query even though the result isn't used there just to try and log things. That means it actually gets called twice each iteration because it's used again by agent_chain. It doesn't seem like a great way of handling things but I'm not sure what a better way would be without rewriting the code I'm trying to evaluate here.
Anyway, I don't see how any of that would be related to the provider errors I'm getting.
Expected behavior I would expect to run the app with recording without errors.
Relevant Logs/Tracebacks
Environment: