[BUG]AzureOpenAI error Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.

[zhongshuai-cao]

Bug Description I am using Azure OpenAI as LLM provider and using Llama_index RAG, and I can instantiate and run question with AOAI provider to get an evaluation with . relevance method successfully. However, when I run the evaluation for my query engine with AOAI provider, I got a lot of screen print of message:

Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result.

To Reproduce Which steps should someone take to run into the same error? A small, reproducible code example is useful here.

from trulens_eval.feedback.provider.openai import AzureOpenAI as AzureOpenAI_eval
from trulens_eval import Tru, Feedback, TruLlama

provider = AzureOpenAI_eval(
    deployment_name=LLM_MODEL,
    endpoint=AZURE_OPENAI_ENDPOINT,
    api_version=AZURE_OPENAI_API_VERSION
)

context = App.select_context(query_engine)

f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons)
    .on(context.collect())
    .on_output()
)

f_answer_relevance = (
    Feedback(provider.relevance)
    .on_input_output()
)

f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons)
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

tru_query_engine_recorder = TruLlama(
    query_engine,
    app_id=app_name,
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
    tru=tru
)

def run_evals(eval_questions, tru_recorder, query_engine):
    for question in eval_questions:
        with tru_recorder as recording:
            response = query_engine.query(question)

run_evals(eval_questions, tru_query_engine_recorder, query_engine)

Expected behavior I should not see this, and I wonder if it was not working as I have coded/planned.

Relevant Logs/Tracebacks Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. /opt/homebrew/Caskroom/miniforge/base/envs/uambot/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py:253: UserWarning: No supporting evidence provided. Returning score only. warnings.warn( Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Callback class OpenAICallback is registered for handling create but there are no endpoints waiting to receive the result. Environment:

• OS: MacOS
• Python Version: 3.11.9
• TruLens version: 0.33.0
• llama-index==0.11.1

Additional context Add any other context about the problem here.

[sfc-gh-dkurokawa]

Hey @zhongshuai-cao , is this still happening to you? I tried this on the newest version of trulens and didn't have this issue.

[zhongshuai-cao]

Thanks for the update! However, I have switched to both the version 1.0.1 and OAI to avoid the issue. Will try that then close it if it worked.

Also I think the newest trulens doesn't support latest llama-index, so I have to downgrade my llama-index and manually patch a history issue fixed in the newer version, while the older version trulens can just use the latest llama-index, which was weird.

[sfc-gh-dkurokawa]

Thanks for the update! However, I have switched to both the version 1.0.1 and OAI to avoid the issue. Will try that then close it if it worked.

Also I think the newest trulens doesn't support latest llama-index, so I have to downgrade my llama-index and manually patch a history issue fixed in the newer version, while the older version trulens can just use the latest llama-index, which was weird.

Okay sounds great. What's the problem with the latest llama-index you're seeing? Can you post another bug for whatever issue you're seeing there and tag me on it?

[zhongshuai-cao]

Oh, It was just pip install requirements conflict warning so I don't treat that as a bug... But I will have to downgrade the version in my requirements