Closed sahusiddharth closed 6 days ago
Describe the bug The function _get_arg_score raises a division by zero error when the refs dictionary is empty. This condition occurs when a tool is called without any arguments, which is a valid scenario, especially during a handoff operation.
Code to Reproduce
from ragas.metrics import ToolCallAccuracy from ragas.dataset_schema import MultiTurnSample from ragas.messages import ToolCall, AIMessage from langchain_openai import ChatOpenAI ragas_trace = [AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='transfer_to_sales_agent', args={})])] sample = MultiTurnSample( user_input=ragas_trace, reference_tool_calls=[ ToolCall(name="transfer_to_sales_agent", args={}) ], ) tool_accuracy_scorer = ToolCallAccuracy() tool_accuracy_scorer.llm = ChatOpenAI(model="gpt-4o-mini") await tool_accuracy_scorer.multi_turn_ascore(sample)
Error trace
File ~/miniconda3/envs/agents/lib/python3.10/site-packages/ragas/metrics/_tool_call_accuracy.py:48, in ToolCallAccuracy._get_arg_score(self, preds, refs, callbacks) 40 if arg in preds: 41 score += await self.arg_comparison_metric.single_turn_ascore( 42 SingleTurnSample( 43 response=str(preds[arg]), reference=str(refs[arg]) 44 ), 45 callbacks, 46 ) ---> 48 return score / len(refs.keys()) ZeroDivisionError: float division by zero
Expected behavior
Describe the bug The function _get_arg_score raises a division by zero error when the refs dictionary is empty. This condition occurs when a tool is called without any arguments, which is a valid scenario, especially during a handoff operation.
Code to Reproduce
Error trace
Expected behavior