explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.33k stars 746 forks source link

ToolCallAccuracy raises ZeroDivisionError when called without any arguments #1684

Closed sahusiddharth closed 6 days ago

sahusiddharth commented 1 week ago

Describe the bug The function _get_arg_score raises a division by zero error when the refs dictionary is empty. This condition occurs when a tool is called without any arguments, which is a valid scenario, especially during a handoff operation.

Code to Reproduce

from ragas.metrics import ToolCallAccuracy
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import ToolCall, AIMessage
from langchain_openai import ChatOpenAI

ragas_trace = [AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='transfer_to_sales_agent', args={})])]

sample = MultiTurnSample(
    user_input=ragas_trace,
    reference_tool_calls=[
        ToolCall(name="transfer_to_sales_agent", args={})
    ],
)

tool_accuracy_scorer = ToolCallAccuracy()
tool_accuracy_scorer.llm = ChatOpenAI(model="gpt-4o-mini")
await tool_accuracy_scorer.multi_turn_ascore(sample)

Error trace

File ~/miniconda3/envs/agents/lib/python3.10/site-packages/ragas/metrics/_tool_call_accuracy.py:48, in ToolCallAccuracy._get_arg_score(self, preds, refs, callbacks)
     40     if arg in preds:
     41         score += await self.arg_comparison_metric.single_turn_ascore(
     42             SingleTurnSample(
     43                 response=str(preds[arg]), reference=str(refs[arg])
     44             ),
     45             callbacks,
     46         )
---> 48 return score / len(refs.keys())

ZeroDivisionError: float division by zero

Expected behavior