Using agents with custom tools completely changes the input if question is asked in different language

[KeshavSingh29]

Feature request

TLDR: Working on using chat-conversational-react-description agent and RetrievalQA as tool to answer queries using vectorDB.

Issue: If question is asked in japanese (Vectordb is in japanese as well), the agent's initial action_input is complete nonsensical (agent automatically translated it to english) which results in wrong final answer.

Request: (1) It would be helpful to somehow manipulate the action_input for agents to not rephrase the input queries when using vector db or prompt support of agents with different languages. (2) It would be more helpful to have someway to see what knowledge the agent is using. Currently, I need to rely on only passing user query to RetrievalQA_chain with return_source_documents=True to check.

Code for reference:

retriever = RetrievalQA.from_chain_type(
    llm=LLM,
    chain_type="stuff",
    retriever=db_retriever,
    return_source_documents=False,
)

retriever_tool_description = """Use this tool when you need to answer specific or game related questions. This tool can also be used for follow up questions from the user.
"""

tools = [
    Tool(
        func=retriever.run, description=retriever_tool_description, name="Game Data DB"
    ),
]

memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    input_key="input",
    output_key="output",
    k=3,
    return_messages=True,
)

conversational_agent = initialize_agent(
    agent="chat-conversational-react-description",
    tools=tools,
    llm=LLM,
    verbose=True,
    max_iterations=2,
    early_stopping_method="generate",
    memory=memory,
    return_intermediate_steps=True,
)

sys_msg = """Your role is to answer the game user's questions in a human-like manner"""

prompt = conversational_agent.agent.create_prompt(system_message=sys_msg, tools=tools)
conversational_agent.agent.llm_chain.prompt = prompt

conversational_agent(input_query)

Output: The top json output is when calling retriever directly on user query. Latter part is output when initializing agent.

Motivation

Its inconvenient to not be able to manipulate what the agents initial action_inputs are. Plus other languages can greatly benefit from such support.

Your contribution

I would like to hear from other people first and then make a PR.

[lawetis]

Yes, my thoughts are with yours.

[KeshavSingh29]

I solved this issue.

1. Its better to use conversational-react-description than chat-conversational-react-description
2. Additionally, its better to change the conversational_agent.agent.llm_chain.prompt to any custom prompt.

Handles any language like a charm. Thanks to the dev. of langchain!

[sindre-bouvet]

@KeshavSingh29 Could you elaborate on the fix on this issue? How did you update the prompt?

[happinessbaby]

@sindre-bouvet These two lines in @KeshavSingh29 's code updated the prompt for me: prompt = conversational_agent.agent.create_prompt(system_message=sys_msg, tools=tools) conversational_agent.agent.llm_chain.prompt = prompt

if you have any input variables in your prompt, you can specify them in create_prompt(....input_variables=["chat_history"]) for example.

[KeshavSingh29]

@sindre-bouvet Its exactly as @happinessbaby mentioned. If you are still unable to fix it, let me know.

[siddhijain47]

Hi @KeshavSingh29 It gives error TypeError: ConversationalAgent.create_prompt() got an unexpected keyword argument 'system_message'

[KeshavSingh29]

@siddhijain47 Can you share your code snippet here? Its difficult to understand where the bug is originating from otherwise.

[siddhijain47]

@siddhijain47 Can you share your code snippet here? Its difficult to understand where the bug is originating from otherwise.

Sure @KeshavSingh29 Here is my code : app.custom_agent = initialize_agent(agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, llm=app.llm, tools=app.tool,

agent_kwargs={"system_message": app.system_message.content},

                             verbose=True, memory=app.memory, handle_parsing_errors=True)
print("print-----",app.custom_agent.agent.llm_chain.prompt.template)

app.prompt = app.custom_agent.agent.create_prompt(system_message=app.system_message.content, tools=app.tool,) app.custom_agent.agent.llm_chain.prompt = app.prompt

[KeshavSingh29]

@siddhijain47 You are using CONVERSATIONAL_REACT_DESCRIPTION type agent. This agent does not have system_message argument in create_prompt function signature.

For this agent you only need to pass only prefix and tools argument to create_prompt. where prefix is analogous to system_message.

As an example:

prompt = app.conversational_agent.agent.create_prompt(
            prefix=prefix,
            tools=app.tools,
        )
app.conversational_agent.agent.llm_chain.prompt = prompt

[siddhijain47]

Hi @KeshavSingh29 , Can you help me out to pass the additional variable/flag while calling RetrievalQA chain agent tool? I don't want my action_input text get converted into english always automatically.

[sebduerr]

Please answer this one!

[KeshavSingh29]

@sebduerr @siddhijain47 I guess your main objective is not to have action_input text to get converted into a specific language automatically. There are two ways I achieved this functionality with almost 100% efficiency.

1. When you define your custom tool (subclass of BaseTool), you can add a more precise description class variable. For example:

   description = f"""
   Tool to answer about something specific. 
   Input should be JSON in the following format: {request_format}"""

   I fiddled around with different request_format and found one with json to be best. eg: request_format = '{{"QUESTION": "<input_question>"}}'

This helps keep the action_input always in the same language.

One issue with this is, since Langchain basically is asking OpenAI to write these dynamic prompts, sometimes the action input can be summarized automatically if your tool input text is too long. To get around it, you will need a lot more fancy stuff like overriding etc.

2. To make the previous point work better, you need to make sure that your main prompt of agent prefix shall mention something like, use only Japanese language to reply.

It should work well. Below is a working case, I can use Japanese text as is for action input.