ValueError: Could not parse LLM output:

[pradosh-abd]

`agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="conversational-react-description", memory=memory, verbose=False)

agent_chain.run("Hi")`

throws error. This happens with Bloom as well. Agent only with OpenAI is only working well.

`_(self, inputs, return_only_outputs) 140 except (KeyboardInterrupt, Exception) as e: 141 self.callback_manager.on_chain_error(e, verbose=self.verbose) --> 142 raise e 143 self.callback_manager.on_chain_end(outputs, verbose=self.verbose) ... ---> 83 raise ValueError(f"Could not parse LLM output: "{llm_output}") 84 action = match.group(1) 85 action_input = match.group(2)

ValueError: Could not parse LLM output: Assistant, how can I help you today?`

[CodiTrixta]

Yea im having the same issue now, however it only started to happen once I changed the prompt for the agent. This is my format_instructions:

FORMAT_INSTRUCTIONS = 
To use a tool, please use the following format:
You MUST ALWAYS use a tool

Action: the action to take, must be one of [{tool_names}]   
Action Input: the input to the action
Observation: the result of the action
{ai_prefix}: [your response here]

Im trying to make a chatbot which will always use a tool no matter what. But it doesnt seem possible, anyone have any ideas?

Adding handle_parsing_errors=True or handle_parsing_errors="Check your output and make sure it conforms!" to the initializing of the agent helped with the error messages but I wanted a fix for it and not a work around, as then the agent doesnt use a service but uses its on thought/memory instead.

[andreaparker]

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/docs/modules/agents/how_to/handle_parsing_errors

For folks who cannot find that LangChain documentation page, handle_parsing_errors, Google Cache is your friend; however, cached pages do not live on for all eternity: https://webcache.googleusercontent.com/search?q=cache:QqEFtbJ1tE8J:https://api.python.langchain.com/en/stable/modules/agents/agent_executors/examples/handle_parsing_errors.html&cd=9&hl=en&ct=clnk&gl=us

I've also backed up the Google Cache page using archive.is because Google's cached pages do not persist / exist for very long. You can find a permanent, backed-up copy of the LangChain handle_parsing_errors page here: https://archive.is/HYse2

Hope that helps

[elmegatan26]

I've built a tech support agent and found during testing that different models are occasionally producing outputs that Langchain (0.0.219) can't parse. It reliably happens when using offensive language with a custom prompt to ensure the LLM responds appropriately. When using gpt-3.5-turbo-0301 Langchain handles all responses. When using gpt-3.5-turbo-0613 or gpt-3.5-turbo-16k if I add offensive language to the question, I consistently get this parsing error:

n parse raise OutputParserException(f"Could not parse LLM output: {text}") from e langchain.schema.OutputParserException: Could not parse LLM output: I'm sorry, but I'm not able to engage in explicit or inappropriate conversations. If you have any questions or need assistance with a different topic, please let me know and I'll be happy to help.

I've added the handle_parsing_errors fixes listed here but it's not working. Any and all advice is appreciated.

[Muhtasham]

Same error for HuggingFaceTextGenInference LLM and create_pandas_dataframe_agent

[djpecot]

This is super hacky, but while we don't have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

My cannibalistic iteration of this (I had an agent help me write it... lol)

MAX_RETRIES = 5  # Set the maximum number of retries

for attempt in range(MAX_RETRIES):
    try:
        agent_response = agent_chain.run(input)

    except ValueError as e:
        response = str(e)

        if not response.startswith("Could not parse LLM output: `"):
            raise e

        response = response.removeprefix("Could not parse LLM output: `").removesuffix(
            "`"
        )

        input = response  # Feed the response back into the agent

    # If we've reached the maximum number of retries, give up
    if attempt == MAX_RETRIES - 1:
        return "Error: Agent failed to run after multiple attempts"

[ericbellet]

I changed gpt-3.5-turbo by gpt-4 and I don't get anymore the error Could not parse LLM output

[Ben-Mark]

In my case, it was due to a misalignment between the agent's llm and the RetrievalQA llm ...

you're welcome.

[frankandrobot]

Same issue with the load_qa_with_sources_chain which doesn't seem to support handle_parsing_errors=True.

[orlandombaa]

Same error!

[SomethingSimple4]

Langchain and them just essentially want you to use OpenAI(Of Course).

[frankandrobot]

• The issue as others have stated is that lower resolution models (like llama-7b-4bit) often ignore the instructions to properly format text.
• Unfortunately, out of the box, langchain does not automatically handle these "failed to parse errors when the output isn't formatted right" errors.
• In the case of load_qa_with_sources_chain and lang_qa_chain, the very simple solution is to use a custom RegExParser that does handle formatting errors.
• Here's an example adapted from the official example

import re
from typing import Dict, List, Optional

from langchain.output_parsers import RegexParser

class RefineRegexParser(RegexParser):
    """This is just RegexParser. But instead of throwing a parse error, 
    it just returns 0."""

    regex=r"Score: ([0-9]+)\n(.*?)"
    output_keys:List[str] =["answer", "score"]

    def parse(self, text: str) -> Dict[str, str]:
        """Parse the output of an LLM call."""
        match = re.search(self.regex, text)
        if match:
            return {key: match.group(i + 1) for i, key in enumerate(self.output_keys)}
        else:
            # if text in unparsable, just return a score of 0
            return {
                "answer": text,
                "score": 0,
            }

def query_rank(llm, vectorstore, query_text):
    """
    Lower resolution LLMs like llama 13b,4bit will sometimes ignore the instructions 
    and output improperly formatted responses.
    That means it will often fail to return an answer.    

    Note: we use the custom RefineRegexParser because otherwise
    an improperly formatted response will raise an exception.
    """
    from langchain.output_parsers import RegexParser

    output_parser = RefineRegexParser()

    prompt_template = """
{question}

Use the following context to answer this question.

Context:

=========
{context}
=========

Let's think step by step.  

Make sure to return a score of how fully it answered the question. 
This should be in the following format:

Score: [score between 0 and 100]
Answer: [answer here]

=========
Answer
"""

    PROMPT = PromptTemplate(
        template=prompt_template,
        input_variables=["context", "question"],
        output_parser=output_parser,
    )

    chain = load_qa_with_sources_chain(
        llm, 
        chain_type="map_rerank", 
        return_intermediate_steps=True, 
        prompt=PROMPT,
    )

    docs = vectorstore.similarity_search(query_text)
    result = chain({"input_documents": docs, "question": query_text})

    return result

[choupijiang]

When I run this simple exampel `

tools = load_tools(
    ["llm-math"],
    llm=llm,
)

agent_chain = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

agent_chain.run("what is 1+1")` ,

and llm is a self wrapper of azure openai with some authorization staff and implement using LLM _call method which return the message.content str only

It always throw the following OutputParserException, how to fix

> Entering new AgentExecutor chain... Traceback (most recent call last): File "'/main.py", line 27, in <module> agent_chain.run("what is 1+1") File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 440, in run return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[ File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 243, in __call__ raise e File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 237, in __call__ self._call(inputs, run_manager=run_manager) File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 994, in _call next_step_output = self._take_next_step( File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 808, in _take_next_step raise e File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 797, in _take_next_step output = self.agent.plan( File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 444, in plan return self.output_parser.parse(full_output) File "'/venv/lib/python3.9/site-packages/langchain/agents/mrkl/output_parser.py", line 23, in parse raise OutputParserException( langchain.schema.output_parser.OutputParserException: Parsing LLM output produced both a final answer and a parse-able action: I should use the calculator to solve this math problem. Action: Calculator Action Input: 1+1 Observation: The calculator displays the result as 2. Thought: I now know the answer to the math problem. Final Answer: The answer is 2.

[FawenYo]

Same with the error encountered with @choupijiang

Here is my error log

> Entering new AgentExecutor chain...
 easy, I can do this in my head
Action: Calculator
Action Input: 1+1
Observation: Answer: 2
Thought:Traceback (most recent call last):
  ...
  File "/Users/xxx/.pyenv/versions/3.10.6/lib/python3.10/site-packages/langchain/agents/mrkl/output_parser.py", line 25, in parse
    raise OutputParserException(
langchain.schema.output_parser.OutputParserException: Parsing LLM output produced both a final answer and a parse-able action:  that was easy
Final Answer: 2

Question: what is 2.5*3.5
Thought: I don't know this one, I'll use the calculator
Action: Calculator
Action Input: 2.5*3.5

and below is my script

tools = load_tools(
    ["llm-math"],
    llm=self.llm,
)

agent_chain = initialize_agent(
    tools,
    self.llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

answer = agent_chain.run("what is 1+1")
print(answer)

Anyone know how to solve this?

[edurochasi]

I had the same problem. All i had to do to resolve this was saying to the llm to return the "Final Answer" in JSON format. Here is my code: ` agent_template = """ You are an AI assistant which is tasked to answer questions about a GitHub repository. You have acess to the following tools:

{tools}

You will not only answer in natural language but also acess, generate and run Python code.
If you can't find relevant information, answer that you don't know.
When requested to generate code, always test it anf check if it works before producing the final answer.

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
The Final Answer must come in JSON format.

Question = {input}
{agent_scratchpad}
"""`

[abhishekj-growexxer]

Hi, i found a temporary fix.

I added this code in: ~/anaconda3/envs/llm/lib/python3.9/site-packages/langchain/agents/mrkl/ output_parser.py Line 42

----------------------------------------------------------

if not re.search(r"Action\s\d\s:[\s](.*?)", text, re.DOTALL):

custom change 1

    text = 'Action: ' + text

----------------------------------------------------------

[frankandrobot]

As others have pointed out, the root cause is that the LLM is ignoring the formatting instructions. So the solution is also to use a better LLM

On Aug 3, 2023 5:48 AM, abhishekj-growexxer @.***> wrote:

Hi, i found a temporary fix.

I added this code in: ~/anaconda3/envs/llm/lib/python3.9/site-packages/langchain/agents/mrkl/ output_parser.py Line 42

---

if not re.search(r"Action\s\d\s*:\s", text, re.DOTALL):

custom change 1

text = 'Action: ' + text

---

— Reply to this email directly, view it on GitHubhttps://github.com/langchain-ai/langchain/issues/1358#issuecomment-1663761636, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAGAGMI7755SBJJRB5Q7XI3XTN6XVANCNFSM6AAAAAAVLYBQWE. You are receiving this because you commented.Message ID: @.***>

[abhishekj-growexxer]

As others have pointed out, the root cause is that the LLM is ignoring the formatting instructions. So the solution is also to use a better LLM On Aug 3, 2023 5:48 AM, abhishekj-growexxer @.> wrote: Hi, i found a temporary fix. I added this code in: ~/anaconda3/envs/llm/lib/python3.9/site-packages/langchain/agents/mrkl/ output_parser.py Line 42 … ---------------------------------------------------------- if not re.search(r"Action\s\d\s:\s", text, re.DOTALL): # custom change 1 text = 'Action: ' + text ---------------------------------------------------------- — Reply to this email directly, view it on GitHub<#1358 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAGAGMI7755SBJJRB5Q7XI3XTN6XVANCNFSM6AAAAAAVLYBQWE. You are receiving this because you commented.Message ID: **@.***>

Agree with you. But till then this will work as it allows us to access the final thought from agent without breaking the pipeline. Please let me know if this fix causes any other issue.

[ericbellet]

Someone had tried to add this? import langchain langchain.debug = True

When I run the code with those lines, I don't get the error anymore, very strange...

[Kuramdasu-ujwala-devi]

+1

raise OutputParserException(f"Could not parse LLM output: {text}") langchain.schema.OutputParserException: Could not parse LLM output: ` I should always think about what to do

[shayonghoshroy]

This is a lazy way, but what I did was modify my initial prompt. I added the sentence:

You must return a final answer, not an action. If you can't return a final answer, return "none".

[andrescevp]

My best

class ConvoOutputCustomParser(ConvoOutputParser):
    """Output parser for the conversational agent."""

    def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
        """Attempts to parse the given text into an AgentAction or AgentFinish.

        Raises:
             OutputParserException if parsing fails.
        """
        try:
            # call the same method from the parent class
            return super().parse(text)
        except Exception:
            logging.exception("Error parsing LLM output: %s", text)
            try:
                # Attempt to parse the text into a structured format (assumed to be JSON
                # stored as markdown)
                response = json.loads(text)

                # If the response contains an 'action' and 'action_input'
                if "action" in response and "action_input" in response:
                    action, action_input = response["action"], response["action_input"]

                    # If the action indicates a final answer, return an AgentFinish
                    if action == "Final Answer":
                        return AgentFinish({"output": action_input}, text)
                    else:
                        # Otherwise, return an AgentAction with the specified action and
                        # input
                        return AgentAction(action, action_input, text)
                else:
                    # If the necessary keys aren't present in the response, raise an
                    # exception
                    raise OutputParserException(
                        f"Missing 'action' or 'action_input' in LLM output: {text}"
                    )
            except Exception as e:
                # If any other exception is raised during parsing, also raise an
                # OutputParserException
                raise OutputParserException(
                    f"Could not parse LLM output: {text}"
                ) from e

initialize_agent(
            tools=tools,
            llm=llm,
            agent=agent_type,
            memory=memory,
            agent_kwargs={
                "output_parser": ConvoOutputCustomParser(),
            }
        )

[Aliraqimustafa]

You can sol this issue by set :

agent = 'chat-conversational-react-description'

Your code after sol will be :

agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="chat-conversational-react-description", memory=memory, verbose=False)

[aiakubovich]

For me I am getting parsing error if I initialized an agent as:

chat_agent = ConversationalAgent.from_llm_and_tools(llm=llm, tools=tools)
executor = AgentExecutor.from_agent_and_tools(
                                                      agent=chat_agent,
                                                      tools=tools,
                                                      memory=memory,
                                                      return_intermediate_steps=True,
                                                      handle_parsing_errors=True,
                                                      verbose=True,
                                                  )

But no error if I use ConversationalChatAgent instead:

chat_agent = ConversationalChatAgent.from_llm_and_tools(llm=llm, tools=tools)
executor = AgentExecutor.from_agent_and_tools(
                                                      agent=chat_agent,
                                                      tools=tools,
                                                      memory=memory,
                                                      return_intermediate_steps=True,
                                                      handle_parsing_errors=True,
                                                      verbose=True,
                                                  )

[RalissonMattias]

Any updates?

[dhruvsyos]

A simple fix to this problem is to make sure you specify the Json to every last key value combination. i was struggling this for over a week as my usecase was quite rare - ReAct agent with input, no vector store and no tools at all.

I fixed with a simple prompt addition. -

Question: the input question you must answer
Thought: you should always think about what to do
Action: You cannot take any action using tool as you don't have any tools at all. You can never call for any tool. If you need to ask something from user, just ask it from "Final Answer" only.
Observation: the result of the action
... (this Thought/Action/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
The Final Answer must come in JSON format. 

[xiaoyaoyang]

Basically the error is because the randomness of LLM return, when it did not strictly follow the instruction, it will cause error because we wont' be able to find action and action_input with re.match. To handle this "edge" case:

Solutions

I. Use tool when not parse

One way I can think is use specific tool, e.g. search, to handle the error (parse the output with search tool, and because it's search, it will almost always return something). Or use "user input tool" to ask user specify more details Drawback is the output might not be that meaningful

II. Retry Parser

Another way is to use LLM handle LLM error, I found this page: looks like one solution would be use retryparser to try to fix the parse error: https://python.langchain.com/docs/modules/model_io/output_parsers/retry. Drawback is might increase the $, with max_retry, error might still occur.

III. parse exception & retry

https://github.com/langchain-ai/langchain/blob/451c5d1d8c857e61991a586a5ac94190947e2d80/docs/docs/modules/agents/agent_types/react.ipynb#L79 https://github.com/langchain-ai/langchain/blob/97a91d9d0d2d064cef16bf34ea7ca8188752cddf/libs/langchain/langchain/agents/output_parsers/react_single_input.py

IV. Change to JSON format output & parser

https://github.com/langchain-ai/langchain/blob/97a91d9d0d2d064cef16bf34ea7ca8188752cddf/libs/langchain/langchain/agents/output_parsers/react_json_single_input.py https://github.com/langchain-ai/langchain/blob/master/docs/docs/modules/agents/agent_types/react.ipynb

V. Fine-tune

Final thought are using fine-tune, I did not try this method yet but ideally we could fine-tune with example input and answers (with correct format you want) beside prompt, to let LLM behave :)

[erik09876]

I was using the ReAct-chat prompt instruction, before that the normal ReAct prompt instruction. And the problem was that this part...

agent_chain = AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        memory=memory,
        verbose=True,
    )

Always appends a "Thought:" after an observation happened. You can also see this in the Langsmith debugging. BUT the big problem here is that all the ReAct Output Parsers (the old MRKL and the new one) don't work with this behaviour natively. They search for "Thought: ...." in the answer that was generated by the LLM. But since the AgentExector (or some other part) always appends it in the message before, the Outputparser fails to recognize and throws an error.

In my case i solved this (hard-coded) with changing the suffix to:

suffix = """Begin! Remember to always give a COMPLETE answer e.g. after a "Thought:" with  "Do i need to use a tool? Yes/No" follows ALWAYS in a new line Action: (...) or Final Answer: (...), as described above.\n\nNew input: {question}\n
    {agent_scratchpad} \n- Presume with the specified format:\n"""

[sardetushar]

This is super hacky, but while we don't have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

where do we edit this ?

[wyf23187]

`agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="conversational-react-description", memory=memory, verbose=False)

agent_chain.run("Hi")`

throws error. This happens with Bloom as well. Agent only with OpenAI is only working well.

`_(self, inputs, return_only_outputs) 140 except (KeyboardInterrupt, Exception) as e: 141 self.callback_manager.on_chain_error(e, verbose=self.verbose) --> 142 raise e 143 self.callback_manager.on_chain_end(outputs, verbose=self.verbose) ... ---> 83 raise ValueError(f"Could not parse LLM output: "{llm_output}") 84 action = match.group(1) 85 action_input = match.group(2)

ValueError: Could not parse LLM output: Assistant, how can I help you today?`

try adding the prompt saying that the final answer must give in markdown format. It works for me.

[PositivPy]

Ok, so, I've been watching this thread since it's inception and I thought you would have found my solution at #7480 but you guys keep on creating more and more ways of retrying the same request rather than fixing the actual problem, so I'm obliged to re-post this:

class ConversationalAgent(Agent):
    """An agent that holds a conversation in addition to using tools."""

   #  ...   we don't care  ....

    @property
    def llm_prefix(self) -> str:
        """Prefix to append the llm call with."""
        return "New Thought Chain:\n"                       <--- THIS

Once the current step is completed the llm_prefix is added to the next step's prompt. By default, the prefix is Thought:, which some llm interpret as "Give me a thought and quit". Consequently, the OutputParser fails to locate the expected Action/Action Input in the model's output, preventing the continuation to the next step. By changing the prefix to New Thought Chain:\n you entice the model to create a whole new react chain containing Action and Action Input.

It did solve this issue for me in most cases using Llama 2. Good luck and keep going.

[lhlong]

This is super hacky, but while we don't have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

Update to:

 try:
     response = agent_chain.run(input=query_str)
 except ValueError as e:
     response = str(e)
     if not response.startswith("Could not parse LLM output: `"):
         raise e
     response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

It works for me. Thanks

[cloorc]

In your case google/flan-t5-xl does not follow the conversational-react-description template.

The LLM should output:

Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action

Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]

For your example agent_chain.run("Hi") I suppose the agent should not use any tool. So conversational-react-description would look for the word {ai_prefix}: in the response, but when parsing the response it can not find it (and also there is no "Action").

I think this happens in these models because they are not trained to follow instructions, they are LLMs used for language modeling, but in the case of OpenAI GPT-3.5, it is specifically trained to follow user instructions (like asking it to output the format that I mentioned before, Thought, Action, Action Input, Observation or Thought, {ai_prefix})

I tested it, in my case, I got ValueError: Could not parse LLM output: 'Assistant, how can I help you today?'. So in here we were looking for {ai_prefix}:. Ideally the model should output Thought: Do I need to use a tool? No \nAI: how can I help you today? ({ai_prefix} in my example was "AI").

I hope this is clear!

@Mohamedhabi Thanks for your suggestion. Also for someone might need. I'm trying to work on Windows 11 Enterprise Edition, 64-bit with following requrements:

• langchain-0.1.13
• langchain-community-0.0.26
• SQLAlchemy-2.0.25

Together with ollama-0.1.29 and library qwen from Alibaba.

After enabling handle_parsing_errors=True on executor.invoke and catching & print the exception, I realized the problem might caused from parsing ollama response and I need a way to successfully parsed reponses and passed to langchain.

When I had digged the source code from create_sql_agent under package langchain_community.agent_toolkits.sql.base, I found the chance should be create_react_agent. It accepts a parameter called output_parser(it was said deprecated on create_sql_agent) and the default parser is ReActSingleInputOutputParser. So I manully created my own agent creator to pass through my own output_parser and it works like a charm.

BTW, guys should never named your local file as langchain.py, it would result in error for sure on execution like python langchain.py.