[Bug]: Get request timeout error every second time I called evaluate method

Bug Description

I was using the faithfulness evaluator to evaluate my custom dataset. However, I got request timeout error every second time I called the evaluate method. Code snippets are attached below:

Version

0.9.27

Steps to Reproduce

ragas_evaluator_gpt4 = Faithfulness(llm=llm_factory("gpt_4"))
service_context_gpt4 = ServiceContext.from_defaults(llm=gpt35)
evaluator_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
from tqdm import tqdm

for key, value in tqdm(test_data.items()):

    query = value["question"]
    answer = value["answer"]
    poor_answer = value["poor_answer"]
    poor_context = value["context_v1"]
    context = value["context_v2"]
    input_dict = {
        "question": query,
        "contexts": context,
        "answer": answer,
    }
    # import ipdb

    # ipdb.set_trace()
    faithfulness = evaluator_gpt4.evaluate(
        query=query, response=answer, contexts=context
    )
    # poor_faithfulness = evaluator_gpt4.evaluate(
    #     query=query, response=poor_answer, contexts=context
    # )
    ragas_faithfulness, intermediate_results = ragas_evaluator_gpt4._score(
        input_dict, callbacks=[]
    )
    # test_data[key]["faithfulness_score_llama_index"] = faithfulness.score
    # test_data[key]["poor_faithfulness_llama_index"] = poor_faithfulness
    test_data[key]["faithfulness_score_ragas"] = ragas_faithfulness
    test_data[key]["intermediate_results_ragas"] = intermediate_results

Relevant Logs/Tracbacks

No response

🤖

Hello @CopyNinja1999! I'm Dosu, a friendly bot here to assist you while we're waiting for a human maintainer. I'm ready to help you debug issues, answer questions, and guide you along the way. Let's solve this problem together!

Based on the information provided, it seems like the timeout error you're experiencing might not be directly related to the FaithfulnessEvaluator class or the aevaluate method. The aevaluate method in the FaithfulnessEvaluator class does not have a parameter named sleep_time_in_seconds, so it's unlikely that this is causing the issue.

The error might be related to the server or application's timeout settings, or it could be due to some other part of your code or setup. Without more specific information about your setup, it's difficult to provide a more precise answer.

To help us better understand the issue, could you please provide more details about your setup? Specifically, it would be helpful to know:

The specifics of your server or application's timeout settings
The complete traceback of the error
Any other relevant information about your setup or the circumstances under which the error occurs

This information will help us identify the cause of the issue and suggest a solution.

In the meantime, you might want to check your server or application's timeout settings and make sure they're not set to a value that could cause a timeout error. If you're using a server or application with a timeout limit for requests, you might want to increase this limit or adjust your code to ensure that requests don't exceed this limit.

You can refer to the FaithfulnessEvaluator class in the LlamaIndex codebase for more information about how the aevaluate method works.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

run-llama / llama_index