[Bug]: ReActAgent seems to be missing arguments in the "Action Input" while parsing LLM's generation

Bug Description

I have a ReActAgent that is capable of using two tools: about_the_user for looking up the user's notes, and look_up_wikipedia for getting answers from Wikipedia articles.

In a particular chain-of-thought, at some stage, the LLM yielded:

Thought: I'll use the information provided by the about_the_user tool to narrow down my search for the drink you enjoy. Action: look_up_wikipedia Action Input: {'pages': ['Coca-Cola'], 'query_str': ''}

Notice that the "Action Input" contains the argument pages. This is seen both from the observability platform of my choice, Arize Phoenix, as well as in the UI of my choice, ChainLit.

However, the ReActAgent pipeline itself somehow ignored the pages argument:

2024-01-28 20:43:01 - HTTP Request: POST http://localhost:1234/v1/chat/completions "HTTP/1.1 200 OK" Thought: I'll use the information provided by the about_the_user tool to narrow down my search for the drink you enjoy. Action: look_up_wikipedia Action Input: {'query_str': ''}

leading to this error:

2024-01-28 20:43:01 - WikipediaReader.load_data() missing 1 required positional argument: 'pages'

Version

0.9.39

Steps to Reproduce

Run my chatbot at this revision. Ask it "Name a type of drink that I enjoy drinking, and then look up the country of origin of that drink."

Relevant Logs/Tracbacks

No response

🤖

Hey there, @tslmy! Good to see you again. I see you're still on the path of making your chatbot the best it can be! 😸

Thank you for providing detailed information about the issue you're experiencing. Based on the context provided, it seems like the ReActAgent is not correctly parsing the "Action Input" arguments.

The method responsible for parsing the "Action Input" arguments for the ReActAgent is _extract_reasoning_step in the ReActAgentWorker class. This method uses the ReActOutputParser to parse the output and extract the reasoning step. If the reasoning step is of type ActionReasoningStep, it is returned along with the message content and a boolean indicating whether the processing is complete. The "Action Input" arguments are part of the ActionReasoningStep object.

The ReActOutputParser class has a parse method that is responsible for parsing the output from the ReAct agent. This method returns an instance of BaseReasoningStep, which can be either ResponseReasoningStep or ActionReasoningStep depending on the output.

If the output contains "Action:", it is assumed that the agent needs to use a tool to answer the question. The extract_tool_use function is used to extract the thought, action, and action input from the output. The action input is then parsed into a dictionary using either json.loads or action_input_parser if the former fails. An ActionReasoningStep instance is then returned with the extracted thought, action, and parsed action input.

Given this, it's possible that there might be an issue with the extract_tool_use function or the action_input_parser in correctly parsing the "Action Input" arguments. It would be helpful to check these functions for any potential issues.

You can find the relevant code in the following files:

I hope this helps in identifying the issue. If you need further assistance, please provide more details about the extract_tool_use function and the action_input_parser, and I'll be glad to help.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Hi @dosu-bot,

Yes, that is helpful. I put a breakpoint in ReActOutputParser.parse, and this is what I saw:

The input parameter, output, had the following content:

"Thought: Great! Now we know what type of drink the user enjoys. Let's look up where coffee originated from.
Action: look_up_wikipedia
Action Input: {'pages': ['coffee'], 'load_kwargs': {}, 'query_str': ''}
"

Note that load_kwargs is a dictionary {}.

Later, extract_tool_use set the variable action_input to "{'pages': ['coffee'], 'load_kwargs': {}". Note that this is NOT a valid JSON string; it is missing a right bracket (}).

The cause of this mistake is that the pattern in extract_tool_use, is matching the right bracket in a non-greedy way, unaware of correct JSON syntax:

Action Input: .*?(\{.*?\})

With action_input malformed, json.loads will throw an exception, and we are now entering action_input_parser (just like you pointed out).

This, however, is again using a regex, which is going to return nothing:

def action_input_parser(json_str: str) -> dict:
    processed_string = re.sub(r"(?<!\w)\'|\'(?!\w)", '"', json_str)
    pattern = r'"(\w+)":\s*"([^"]*)"'
    matches = re.findall(pattern, processed_string)
    return dict(matches)

That's the root cause of the problem.

Any suggestions on how to fix it?

🤖

Thank you for the detailed analysis, it's very helpful in understanding the issue. The problem seems to be with the regular expression used in the extract_tool_use function. The non-greedy matching is causing the JSON string to be malformed, which is why the pages argument is not being passed correctly.

To fix this, you could modify the regular expression to correctly match the entire JSON string. Here's an example of how you could do this:

# Old pattern
# pattern = r"Action Input: .*?(\{.*?\})"

# New pattern
pattern = r"Action Input: .*?(\{.*\})"

This new pattern should correctly match the entire JSON string, including the right bracket.

As for the action_input_parser function, it seems to be designed to handle a specific format of JSON strings. If your JSON string doesn't match this format, the function will not return the expected result. You might need to modify this function to handle the format of JSON strings that you're working with.

Here's an example of how you could modify the action_input_parser function:

def action_input_parser(json_str: str) -> dict:
    processed_string = re.sub(r"(?<!\w)\'|\'(?!\w)", '"', json_str)
    try:
        return json.loads(processed_string)
    except json.JSONDecodeError:
        pattern = r'"(\w+)":\s*"([^"]*)"'
        matches = re.findall(pattern, processed_string)
        return dict(matches)

This modified function first tries to parse the JSON string using json.loads. If this fails, it falls back to the original method of using a regular expression to parse the string.

Please try these modifications and let me know if they solve your issue. If not, I'll be happy to help you further troubleshoot this problem.

Sources

llama_index/readers/wikipedia.py

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

run-llama / llama_index