ValueError: Could not parse LLM output:

[pradosh-abd]

`agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="conversational-react-description", memory=memory, verbose=False)

agent_chain.run("Hi")`

throws error. This happens with Bloom as well. Agent only with OpenAI is only working well.

`_(self, inputs, return_only_outputs) 140 except (KeyboardInterrupt, Exception) as e: 141 self.callback_manager.on_chain_error(e, verbose=self.verbose) --> 142 raise e 143 self.callback_manager.on_chain_end(outputs, verbose=self.verbose) ... ---> 83 raise ValueError(f"Could not parse LLM output: "{llm_output}") 84 action = match.group(1) 85 action_input = match.group(2)

ValueError: Could not parse LLM output: Assistant, how can I help you today?`

[jamespacileo]

same here can't figure it out

[Mohamedhabi]

In your case google/flan-t5-xl does not follow the conversational-react-description template.

The LLM should output:

Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action

Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]

For your example agent_chain.run("Hi") I suppose the agent should not use any tool. So conversational-react-description would look for the word {ai_prefix}: in the response, but when parsing the response it can not find it (and also there is no "Action").

I think this happens in these models because they are not trained to follow instructions, they are LLMs used for language modeling, but in the case of OpenAI GPT-3.5, it is specifically trained to follow user instructions (like asking it to output the format that I mentioned before, Thought, Action, Action Input, Observation or Thought, {ai_prefix})

I tested it, in my case, I got ValueError: Could not parse LLM output: 'Assistant, how can I help you today?'. So in here we were looking for {ai_prefix}:. Ideally the model should output Thought: Do I need to use a tool? No \nAI: how can I help you today? ({ai_prefix} in my example was "AI").

I hope this is clear!

[benthecoder]

Am having the same issue

My code is here

@st.cache_resource
def create_tool(_index, chosen_pdf):
    tools = [
        Tool(
            name=f"{chosen_pdf} index",
            func=lambda q: str(_index.query(q)),
            description="Useful to answering questions about the given file",
            return_direct=True,
        ),
    ]

    return tools

@st.cache_resource
def create_agent(chosen_class, chosen_pdf):
    memory = ConversationBufferMemory(memory_key="chat_history")
    llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo")

    index = get_index(chosen_class, chosen_pdf)
    tools = create_tool(index, chosen_pdf)

    agent = initialize_agent(
        tools, llm, agent="conversational-react-description", memory=memory
    )

    return agent

def query_gpt_memory(chosen_class, chosen_pdf, query):

    agent = create_agent(chosen_class, chosen_pdf)

    res = agent.run(input=query)

    st.session_state.memory = agent.memory.buffer

    return res

error output

  File "/Users/benedictneo/fun/ClassGPT/app/utils.py", line 158, in query_gpt_memory
    res = agent.run(input=query)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 268, in run
    return self(kwargs)[self.output_keys[0]]
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 168, in __call__
    raise e
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 165, in __call__
    outputs = self._call(inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 503, in _call
    next_step_output = self._take_next_step(
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 406, in _take_next_step
    output = self.agent.plan(intermediate_steps, **inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 102, in plan
    action = self._get_next_action(full_inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 64, in _get_next_action
    parsed_output = self._extract_tool_and_input(full_output)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/conversational/base.py", line 84, in _extract_tool_and_input
    raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `Thought: Do I need to use a tool? No

However, I still get a response even with this value error

[linbojin]

+1

[gcsun]

+1

[eriktlu]

I'm having the same problem with OpenAIChat llm

[hm-ca]

Yes same issue here

[Arttii]

Just to understand clearly, is there a correct way of going about this? I am also getting the same error even if i use the chat-* type agents.

Or should we wait for an update?

[bcarsley]

just boosting this thread -- this is a common issue in my builds with langchain, but I also utilize promptlayer, so I can see that the outputs are indeed parsable to the API that routes my convo logs...clearly something is wrong and I have yet to find a full-proof solution to avoiding this.

[sundar7D0]

+1 This seems to a common issue with chat agents which are the need of the hour!

[racinger]

+1 same issue here

[gambastyle]

same here...

[tomsib2001]

I was able to fix this locally by simply calling the LLM again when there is a Parse error. I am sure my code is not exactly in the spirit of langchain, but if anyone wants to take the time to review my branch https://github.com/tomsib2001/langchain/tree/fix_parse_LLL_error (it's a POC, not a finished MR) and tell me:

• either that there is a fundamental reason not to do things this way
• how to make my code more in line with the standards of langchain I would be grateful. I am happy to help as I'm getting a lot of value and fun from langchain, but I'm fairly new to contributing to large open source projects, so please bear with me.

[iraadit]

I have the same problem

[CymDanus]

Same problem

[martinv-bits2b]

I have a same problem. Don't know what can be the reason, when the output that cause the error contains well formulated answer.

[franciscoescher]

This is super hacky, but while we don't have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

[noobmaster19]

Might be anecdotal, but I think GPT does better with JSON-type formatting, maybe that will help with the formating issues? I made a couple of changes on my local copy of langchain, seems to work much more reliably

[nipunj15]

@noobmaster19 can you share what changes you made to make things more reliable?

[alexiri]

Are there any HuggingFace models that work as an agent, or are we forced to use OpenAI?

[Saik0s]

It is possible to pass output parser to the agent executor. Here is how I did:

class NewAgentOutputParser(BaseOutputParser):
    def get_format_instructions(self) -> str:
        return FORMAT_INSTRUCTIONS

    def parse(self, text: str) -> Any:
        print("-" * 20)
        cleaned_output = text.strip()
        # Regex patterns to match action and action_input
        action_pattern = r'"action":\s*"([^"]*)"'
        action_input_pattern = r'"action_input":\s*"([^"]*)"'

        # Extracting first action and action_input values
        action = re.search(action_pattern, cleaned_output)
        action_input = re.search(action_input_pattern, cleaned_output)

        if action:
            action_value = action.group(1)
            print(f"First Action: {action_value}")
        else:
            print("Action not found")

        if action_input:
            action_input_value = action_input.group(1)
            print(f"First Action Input: {action_input_value}")
        else:
            print("Action Input not found")

        print("-" * 20)
        if action_value and action_input_value:
            return {"action": action_value, "action_input": action_input_value}

        # Problematic code left just in case
        if "```json" in cleaned_output:
            _, cleaned_output = cleaned_output.split("```json")
        if "```" in cleaned_output:
            cleaned_output, _ = cleaned_output.split("```")
        if cleaned_output.startswith("```json"):
            cleaned_output = cleaned_output[len("```json"):]
        if cleaned_output.startswith("```"):
            cleaned_output = cleaned_output[len("```"):]
        if cleaned_output.endswith("```"):
            cleaned_output = cleaned_output[: -len("```")]
        cleaned_output = cleaned_output.strip()
        response = json.loads(cleaned_output)
        return {"action": response["action"], "action_input": response["action_input"]}
        # end of problematic code

def make_chain():
    memory = ConversationBufferMemory(
        memory_key="chat_history", return_messages=True)

    agent = ConversationalChatAgent.from_llm_and_tools(
        llm=ChatOpenAI(), tools=[], system_message=SYSTEM_MESSAGE, memory=memory, verbose=True, output_parser=NewAgentOutputParser())

    agent_chain = AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        memory=memory,
        verbose=True,
    )
    return agent_chain

[gambastyle]

@Saik0s I am interested in this approach, can you explain what is the SYSTEM_MESSAGE, have you changed it? and can you show how the output looks? does it also work with zero-shot agent?

[Saik0s]

@gambastyle I did some modifications to system message, but that is not related to the problem. You can find default system message here. I just wanted to show my approach to fix problem by introducing upgraded output parser NewAgentOutputParser

[tiagoefreitas]

Everyone seems to have missed klimentij's fix in this thread? It fixes it for me, even though the best solution would be to fix the agent to use a series of chains like he did privately.

Having a prompt doing more than one thing and parsing the output is a bad idea with LLMs, that is why chains exist and are not being properly used in this agent.

See code and comments here: https://github.com/hwchase17/langchain/pull/1707

@Saik0s did you check klimentij's parser changes and do you changes compare?

[Saik0s]

@tiagoefreitas I checked it, his approach is similar to what I did, and he did it directly in the library code. I would even say that his approach is better, because there is no regular expressions in it 😅

[eleijonmarck]

super duper hacky solution:

if __name__ == "__main__":
    try:
        response = agent.run(
           "Could you fix this issue please?"
        )
    except Exception as e:
        response = str(e)
        if "Could not parse LLM output: `" not in response:
            raise e

        match = re.search(r"`(.*?)`", response)

        if match:
            last_output = match.group(1)
            print("Last output:", last_output)
        else:
            print("No match found")

 # Last output: hopefully it will be fixed soon!

[kjain25]

try: response = agent_chain.run(input=query_str) except ValueError as e: response = str(e) if not response.startswith("Could not parse LLM output: "): raise e response = response.removeprefix("Could not parse LLM output:").removesuffix("`")

Where do I paste this? To make the agent do this?

[haqqibrahim]

I am still getting the same error even after applying the proposed solution, can any one please help me?

[tylermaran]

I'm getting this whenever I try to add an ai_prefix to ConversationalAgent.create_prompt(). This one is really surprising to me. It's just changing out the string value of "AI"

def create_prompt(
        cls,
        tools: Sequence[BaseTool],
        prefix: str = PREFIX,
        suffix: str = SUFFIX,
        format_instructions: str = FORMAT_INSTRUCTIONS,
        ai_prefix: str = "AI",
        human_prefix: str = "Human",
        input_variables: Optional[List[str]] = None,
    ) -> PromptTemplate:

With the prefix:

prompt = ConversationalAgent.create_prompt(
    tools,
    prefix,
    suffix,
    ai_prefix="Dude",
    input_variables=["input", "chat_history", "agent_scratchpad"],
)

Output

New Question:  hello
Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?Traceback (most recent call last):
raise OutputParserException(f"Could not parse LLM output: `{text}`")
langchain.schema.OutputParserException: Could not parse LLM output: `Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?`

Without Prefix

New Question:  hello
Thought: Do I need to use a tool? No
AI: Hello! How can I help you with your Medicare questions today?

[talhaanwarch]

in short there is not a single huggingface model, that is not throwing this error?

[alexiri]

I have yet to find a model that could replace OpenAI in Langchain. I've tried LLama 30B, 13B and 7B, Vicuna 13B, Dolly v2 12B, Alpaca-Lora 7B, GPT NeoXT 20B, oasst-sft-6-llama-30b, Flan T5 XXL, BloomZ 7b1... None of them are even close to working, it's really disappointing.

[talhaanwarch]

I have yet to find a model that could replace OpenAI in Langchain. I've tried LLama 30B, 13B and 7B, Vicuna 13B, Dolly v2 12B, Alpaca-Lora 7B, GPT NeoXT 20B, oasst-sft-6-llama-30b, Flan T5 XXL, BloomZ 7b1... None of them are even close to working, it's really disappointing.

does all throw the same error? ValueError: Could not parse LLM output

[alexiri]

It depends on what you're doing, but that error is just a symptom. The cause is the LLM now giving the output it's been told to give. In many cases it was just ignoring instructions, giving nonsense answers, getting confused...

Things like this, for example: (this is Vicuna 13b with a chain with tools, and the input was "Hi!")

> Entering new AgentExecutor chain...
Thought: I should greet the human in a friendly and witty manner.
Action: CERN
Action Input: "What experiments are currently being conducted at CERN?"

Observation:{'question': 'What experiments are currently being conducted at CERN?', 'answer': '1. What is the text:\n\n\n\nsources,\nand\nand\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor\nor answer:\nor\n\n\n\n\n\n\n\nor\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nor answer\nor\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nwhich answer.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nor\nor, it, it,\nor,', 'sources': ''}
Thought: I have gathered all the information I need to reply to the human.
Answer: <completely made up answer>

The output was parseable, at least, but complete nonsense.

[vinodvarma24]

+1. Please solve this issue.

[tjaffri]

I had some luck (but not perfect) with google/flan-ul2 using few shot prompting. Here's my PR if people want to upvote / give feedback: https://github.com/hwchase17/langchain/pull/3636

[sd1219]

Two potential fixes are:

1. If you're using a tool: return_direct=True
2. If not, then experiment with the prefix prompt by modifying it a bit for your specific task.

[Devonance]

@sd1219 Sadly doing number 1 on your list breaks the chain for me and it only does one thought and one observation (I'm using the ReAct logic + Langchain + GPT-4 API).

Tool.from_function(
        func=docstore.search,
        name="Search",
        description="Useful for when you need to ask with search.",
        return_direct=True
    )

Only @franciscoescher 's method worked for me. The end answer would be given before the awful error codes though.

[skeretna]

I have the same problem too :(

[Chesterguan]

Hi I tried to modify the outputParser() for the ConversationChatAgent to fix the issue, however, I met a new issue as AttributeError: 'str' object has no attribute 'tool'

Seems like I cannot parse the names of Tools.

below are my codes:

class CustomizedOutputParser(AgentOutputParser):
    """
    A parser class to extract action and action_input from LLM 
    output text. It processes the text by removing code block 
    delimiters and extracts the necessary information.
    """
    def get_format_instructions(self) -> str:
        return FORMAT_INSTRUCTIONS

    def parse(self, text: str) -> Dict[str, str]:
        cleaned_output = text.strip()

        if "```json" in cleaned_output:
            _, cleaned_output = cleaned_output.split("```json")
        if cleaned_output.startswith("```json"):
            cleaned_output = cleaned_output[len("```") :]
        if cleaned_output.endswith("```"):
            cleaned_output = cleaned_output[: -len("```")]

        cleaned_output = cleaned_output.strip()
        try:
            response = json.loads(cleaned_output)
        except json.JSONDecodeError:
            response = {}
            cleaned_output = cleaned_output.replace('\n', '')

            try:
                response["action"] = cleaned_output.split('"action": "', 1)[1].split('",', 1)[0].strip()
                response["action_input"] = cleaned_output.split('"action_input": "', 1)[1].split('"}', 1)[0].strip()
            except IndexError:
                raise ValueError("Invalid input format. Unable to extract 'action' and 'action_input' from the text.")

        return {"action": response["action"], "action_input": response["action_input"]}

class ChatBot:
    def __init__(self, memory, agent_chain):
        self.memory = memory
        self.agent_chain = agent_chain

def create_chatbot(model_name, seed_memory=None):
    search = GoogleSearchAPIWrapper()
    arXiv = ArxivAPIWrapper()
    python_repl = PythonREPL()
    tools = [
        Tool(
            name="Current Search",
            func=search.run,
            description="useful for all question that asks about live events",
        ),
        Tool(
            name="Topic Search",
            func=search.run,
            description="useful for all question that are related to a particular topic, product, concept, or service",
        ),
        Tool(
            name="Archiv Paper Search",
            func=arXiv.run,
            description="useful for all question that asks about scientific papers",
        ), 
        Tool(
            name="Python REPL",
            func=python_repl.run,
            description="useful for all question that asks about Python codes",
        ), 

    ]
    # add gradio tools
    tools.extend([StableDiffusionTool().langchain, BarkTextToSpeechTool().langchain])

    chat = ChatOpenAI(temperature=0, model_name=model_name)
    memory = seed_memory if seed_memory is not None else ConversationBufferMemory(memory_key="chat_history", input_key="input", output_key="output", return_messages=True)

    agent = ConversationalChatAgent.from_llm_and_tools(
        llm=chat, tools=tools, memory=memory, verbose=True, output_parser=CustomizedOutputParser())

    agent_chain = AgentExecutor.from_agent_and_tools(
        tools=tools,
        agent=agent,
        verbose=True, memory=memory, return_intermediate_steps=True)
    return ChatBot(memory, agent_chain)

[okoliechykwuka]

I encountered this error while using Openai "gpt-3.5-turbo" model. The Agent was unable to use the LLM to extract the needed info from the vectorstore, so I switched to the default model, and the error was gone

from llm = OpenAI(model_name = "gpt-3.5-turbo",temperature=0)

to llm = OpenAI(temperature=0)

[shellyfung]

i have the same problem and i fixed this by add prompt with "please give me plain text without any punctuation mark"

[Willis75]

Please check this: https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/handle_parsing_errors.html

Worked for me

[Muzammiljethwa]

same issue here in langchain agent Can anyone specify, what is this issue caused by?

[lezgoverci]

I encountered this error while using Openai "gpt-3.5-turbo" model. The Agent was unable to use the LLM to extract the needed info from the vectorstore, so I switched to the default model, and the error was gone

from llm = OpenAI(model_name = "gpt-3.5-turbo",temperature=0)

to llm = OpenAI(temperature=0)

this solved my problem

[icedlattesuki]

I got this parsing error when usingconversational-react-description agent and gpt-3.5-turbo model.

After two ReAct rounds, the LLM thought not using any tool, but the response didn't include the prefix and the AI response.

it looks like this:

langchain.schema.OutputParserException: Could not parse LLM output: `Do I need to use a tool? No`

I tried to update the instruction prompt to enforce the LLM returns the formatted text. This works for me.

my code:

    agent = initialize_agent(
        tools,
        llm,
        agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
        verbose=True,
        memory=get_memory(request.user_id),
        # pass custom instruction prompt here
        agent_kwargs={"format_instructions": FORMAT_INSTRUCTIONS},
    )

Adding (the prefix of "Thought: " and "{ai_prefix}: " are must be included) into the original prompt. (just ignore the '\')

FORMAT_INSTRUCTIONS = """To use a tool, please use the following format:

\```
Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
\```

When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the following format(the prefix of "Thought: " and "{ai_prefix}: " are must be included):

\```
Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]
\```"""

Hope it could help :)

[davletovb]

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/docs/modules/agents/how_to/handle_parsing_errors

[az077xtz]

Same for me

[Hamas-ur-Rehman]

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/handle_parsing_errors.html

For anyone facing this issue try this method worked for me and good documentation if you want custom handling Thanks @davletovb

[marcoslneves]

I was getting the same error but this code worked for me ( langchain==0.0.196 ) The fix was to add "Action: " at the end of the suffix to tell the Agent to use the result action and not the answer text. A tool expects to excute an action and when an answer is produced it gets confused.

# memory
message_history = DynamoDBChatMessageHistory(
    table_name="<table_name>", session_id="1"
)

memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    chat_memory=message_history,
    return_messages=True,
    k=3,
)

# LLMChain
prefix = """Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:"""

suffix = """Begin!

{chat_history}

Question: {input}
Thought: {agent_scratchpad}
Action: """

prompt = ZeroShotAgent.create_prompt(
    tools,
    prefix=prefix,
    suffix=suffix,
    input_variables=["input", "chat_history", "agent_scratchpad"],
)
llm_chain = LLMChain(llm=llm, prompt=prompt, verbose=True)

# Agent
agent = ZeroShotAgent(
    llm_chain=llm_chain,
    tools=[]
)
agent_chain = AgentExecutor.from_agent_and_tools(
    agent=agent,
    tools=tools,
    verbose=True,
    memory=memory,
    # return_intermediate_steps=True,
)

answer = agent_chain.run("My question with memory")

[Prasanna-iq]

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/handle_parsing_errors.html

Link is not working