Langchain MRKL Agent not giving useful Final Answer

[aju22]

Discussed in https://github.com/hwchase17/langchain/discussions/7423

^{Originally posted by **aju22** July 9, 2023} Here is the code I'm using for initializing a Zero Shot ReAct Agent with some tools for fetching relevant documents from a vector database: ``` chat_model = ChatOpenAI( model_name="gpt-3.5-turbo", temperature="0", openai_api_key=openai_api_key, streaming=True, # verbose=True) llm_chain = LLMChain(llm=chat_model, prompt=prompt) agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True, handle_parsing_errors=True) agent_chain = AgentExecutor.from_agent_and_tools( agent=agent, tools=tools, verbose=True, memory=memory ) ``` However when I query for a response. ``` query = "Can you explain a use case example of chain of thought prompting in detail?" res = agent_chain(query) ``` This is the response I get back: ``` > Entering new chain... Thought: The question is asking for a detailed explanation of a use example of chain-of-thought prompting. Action: Lookup from database Action Input: "use example of chain-of-thought prompting" Observation: Sure! Here's an example of chain-of-thought prompting: Let's say we have a language model that is trained to solve math word problems. We want to use chain-of-thought prompting to improve its reasoning abilities. The prompt consists of triples: input, chain of thought, output. For example: Input: "John has 5 apples." Chain of Thought: "If John gives 2 apples to Mary, how many apples does John have left?" Output: "John has 3 apples left." In this example, the chain of thought is a series of intermediate reasoning steps that lead to the final output. It helps the language model understand the problem and perform the necessary calculations. By providing these chain-of-thought exemplars during training, the language model learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference. This approach of chain-of-thought prompting has been shown to improve the performance of language models on various reasoning tasks, including arithmetic, commonsense, and symbolic reasoning. It allows the models to decompose complex problems into manageable steps and allocate additional computation when needed. Overall, chain-of-thought prompting enhances the reasoning abilities of large language models and helps them achieve state-of-the-art performance on challenging tasks. Thought:I have provided a detailed explanation and example of chain-of-thought prompting. Final Answer: Chain-of-thought prompting is a method used to improve the reasoning abilities of large language models by providing demonstrations of chain-of-thought reasoning as exemplars in prompting. It involves breaking down multi-step problems into manageable intermediate steps, leading to more effective reasoning and problem-solving. An example of chain-of-thought prompting is providing a language model with a math word problem prompt consisting of an input, chain of thought, and output. By training the model with these exemplars, it learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference. This approach has been shown to enhance the performance of language models on various reasoning tasks. > Finished chain. ``` As you can observe, The model has a very thorough and exact answer in it's observation. However in the next thought, the model thinks it is done providing a detailed explanation and example to the human. So the final answer is just some basic information, not really answering the question in necessary detail. I feel like somewhere in the intermediate steps, the agent thinks it has already answered to the human, and hence just does not bother to give that as the final answer. Can someone please help me figure out, how can I make the model output it's observation as the final answer. Or to stop making the model assume it has already answered the question to the human. Will playing around with the prompt template work?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're experiencing an issue where the final answer provided by the LangChain MRKL Agent is not as detailed as the observation. It seems like the model is assuming it has already provided a sufficient answer, and as a result, the final answer lacks the necessary detail.

One potential solution to this issue could be to adjust the Thought before the Final Answer in the prompt template. Instead of the model thinking "I now know the final answer", you could change this to something like "I have gathered detailed information to answer the question". This might encourage the model to provide a more comprehensive final answer.

Here's how you can modify the prompt template:

FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I have gathered detailed information to answer the question
Final Answer: the final answer to the original input question"""

Please note that this is a heuristic approach and might not always lead to the desired results. The effectiveness of this approach would depend on the specific behavior of the language model being used.

You also have the option to create a custom LLMChain and feed agents a self-defined prompt template, not restricted to the prompt generated by the create_prompt function, as long as it meets the agent's requirements. For ZeroShotAgent, you need to ensure that there is a string starting with "Action:" and a following string starting with "Action Input:", and both should be separated by a newline.

Furthermore, you can override the create_prompt method in a subclass to customize the prompt creation process. For example, you could modify the format_instructions argument to change the instructions given to the model. This could potentially influence the model's behavior and lead to more detailed final answers.

Lastly, you can modify the default prefix, format instructions, and suffix used in the prompt in the prompt.py file to further customize the behavior of your agent. For example, you could modify the FORMAT_INSTRUCTIONS to change the instructions given to the model, or you could modify the PREFIX to provide additional context to the model. This could potentially influence the model's behavior and lead to more detailed final answers.

Please remember that these are potential solutions and might not always work as expected. If you find a solution that works well, and you believe it could be beneficial to other users, feel free to make a pull request. If you're still experiencing issues, I would recommend checking if you're using the latest version of LangChain, as updates often include bug fixes and improvements.

Sources

• langchain/agents/mrkl/prompt.py
• docs/extras/modules/agents/how_to/custom_mrkl_agent.ipynb
• langchain/agents/mrkl/base.py
• langchain/agents/conversational/prompt.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[Khalidzo]

@aju22 hey, I hope you have figured out how to solve this issue since I'm also dealing with it. I honestly didn't fully understand the solution that was given by the bot, so would you please share your solution?

[aju22]

@Khalidzo For my particular use case, I wanted the model to act sort of a like a teacher and give bullet point answers for easy understanding.

With a bit of hacks and "prompt engineering", I get fairly good answers and also worked out a way around most frustrating LLM Output Parse Errors, and it works 99% of the time.

Please check out the final implementation in my project at: DocumentGPT You will find the agent declared in def get_agent(self)

P.S: It does consume a good amount of tokens to get it working.

[limaoyi1]

Great job! @aju22 This can effectively extract information from the Observations.