StructuredChatOutputParser too Lenient with Final Answers

[devstein]

At present, StructuredChatOutputParser assumes that if there is not matching ```, then the full text is the "Final Answer". The issue is that in some cases (due to truncation, etc), the output looks like (sic):

I have successfully navigated to asdf.com and clicked on the sub pages. Now I need to summarize the information on each page. I can use the `extract_text` tool to extract the information on each page and then provide a summary of the information. 
Action:

[ { "action": "click_element", "action_input": {"selector": "a[href='https://www.asdf.com/products/widgets/']"} }, { "action": "extract_text", "action_input": {}

In these cases (such as when the text "Action:" and/or "```" appear), it may be safer to have fallback actions that re-tries rather than just assuming this is the final answer.

[dosubot[bot]]

File: langchain/agents/structured_chat/output_parser.py

To update the file and address the issue, I would modify the StructuredChatOutputParser class in the langchain/agents/structured_chat/output_parser.py file. Specifically, I would update the parse method to check for the presence of "Action:" and/or "```" in the text before assuming it's the final answer.

Here's the updated parse method:

def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
    try:
        action_match = re.search(r"```(.*?)```?", text, re.DOTALL)
        if action_match is not None:
            response = json.loads(action_match.group(1).strip(), strict=False)
            if isinstance(response, list):
                logger.warning("Got multiple action responses: %s", response)
                response = response[0]
            if response["action"] == "Final Answer":
                return AgentFinish({"output": response["action_input"]}, text)
            else:
                return AgentAction(
                    response["action"], response.get("action_input", {}), text
                )
        else:
            # Check for the presence of "Action:" and/or "```" in the text
            if "Action:" in text or "```" in text:
                raise OutputParserException(f"Could not parse LLM output: {text}")
            return AgentFinish({"output": text}, text)
    except Exception as e:
        raise OutputParserException(f"Could not parse LLM output: {text}") from e

This change ensures that if "Action:" and/or "``" are present in the text, anOutputParserException` is raised, allowing for fallback actions to be implemented. File: langchain/agents/conversational_chat/output_parser.py

To update the file and address the issue mentioned in the request, I would modify the parse method in the ConvoOutputParser class to include a check for the presence of "Action:" and/or "```" in the output. If either of these is found, we can raise an exception or implement a fallback mechanism to re-try the parsing. Here's how I would update the method:

from __future__ import annotations

import json
from typing import Union

from langchain.agents import AgentOutputParser
from langchain.agents.conversational_chat.prompt import FORMAT_INSTRUCTIONS
from langchain.schema import AgentAction, AgentFinish

class ConvoOutputParser(AgentOutputParser):
    def get_format_instructions(self) -> str:
        return FORMAT_INSTRUCTIONS

    def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
        cleaned_output = text.strip()
        if "```json" in cleaned_output:
            _, cleaned_output = cleaned_output.split("```json")
        if "```" in cleaned_output:
            cleaned_output, _ = cleaned_output.split("```")
        if cleaned_output.startswith("```json"):
            cleaned_output = cleaned_output[len("```json") :]
        if cleaned_output.startswith("```"):
            cleaned_output = cleaned_output[len("```") :]
        if cleaned_output.endswith("```"):
            cleaned_output = cleaned_output[: -len("```")]
        cleaned_output = cleaned_output.strip()

        # Check for the presence of "Action:" and/or "```" in the output
        if "Action:" in cleaned_output or "```" in cleaned_output:
            # Raise an exception or implement a fallback mechanism to re-try the parsing
            raise ValueError("Output parsing failed due to incomplete or truncated output")

        response = json.loads(cleaned_output)
        action, action_input = response["action"], response["action_input"]
        if action == "Final Answer":
            return AgentFinish({"output": action_input}, text)
        else:
            return AgentAction(action, action_input, text)

This update adds a check for the presence of "Action:" and/or "``" in the cleaned_output. If either is found, aValueErroris raised with a message indicating that the output parsing failed due to incomplete or truncated output. You can replace theraise ValueError` line with a fallback mechanism to re-try the parsing if desired.