dosubot[bot]
commented
9 months ago 🤖
The issue you're experiencing seems to be due to the BatchEvalRunner not properly handling exceptions that occur during the evaluation process. This can be fixed by adding a try/except block around the evaluation calls in the aevaluate_response_strs, aevaluate_responses, and aevaluate_queries methods.
Here is the updated code for the aevaluate_response_strs method:
async def aevaluate_response_strs(
self,
queries: Optional[List[str]] = None,
response_strs: Optional[List[str]] = None,
contexts_list: Optional[List[List[str]]] = None,
**eval_kwargs_lists: List,
) -> Dict[str, List[EvaluationResult]]:
"""Evaluate query, response pairs.
This evaluates queries, responses, contexts as string inputs.
Can supply additional kwargs to the evaluator in eval_kwargs_lists.
Args:
queries (Optional[List[str]]): List of query strings. Defaults to None.
response_strs (Optional[List[str]]): List of response strings.
Defaults to None.
contexts_list (Optional[List[List[str]]): List of context lists.
Defaults to None.
**eval_kwargs_lists (Dict[str, Any]): Dict of lists of kwargs to
pass to evaluator. Defaults to None.
"""
queries, response_strs, contexts_list = self._validate_and_clean_inputs(
queries, response_strs, contexts_list
)
for k in eval_kwargs_lists:
v = eval_kwargs_lists[k]
if not isinstance(v, list):
raise ValueError(
f"Each value in eval_kwargs must be a list. Got {k}: {v}"
)
eval_kwargs_lists[k] = self._validate_and_clean_inputs(v)[0]
# run evaluations
eval_jobs = []
for idx, query in enumerate(cast(List[str], queries)):
response_str = cast(List, response_strs)[idx]
contexts = cast(List, contexts_list)[idx]
eval_kwargs = self._get_eval_kwargs(eval_kwargs_lists, idx)
for name, evaluator in self.evaluators.items():
eval_jobs.append(
eval_worker(
self.semaphore,
evaluator,
name,
query=query,
response_str=response_str,
contexts=contexts,
eval_kwargs=eval_kwargs,
)
)
try:
results = await self.asyncio_mod.gather(*eval_jobs)
except Exception as e:
print(f"An error occurred during evaluation: {e}")
results = []
# Format results
return self._format_results(results)You would need to add similar try/except blocks to the aevaluate_responses and aevaluate_queries methods as well. This will ensure that any exceptions that occur during the evaluation process are caught and handled, preventing the loss of all results.
You can find the original implementation of the BatchEvalRunner class in the LlamaIndex repository.
Please let me know if this solution works for you or if you need further assistance.
peguerosdc
justinzyw
chrispe2
Bug Description
When running
BatchEvalRunneron a large amount of test cases and metrics, if one of them fails (I found it common to happen withGuidelineEvaluatordue to its unreliable JSON parsing), the exception is not caught and you lose all the other results, resulting in a loss of time and money.Version
0.10.4
Steps to Reproduce
It doesn't always fail, so you might need to run it a couple of times.
Relevant Logs/Tracbacks