SystemMessage are not considered while creating AgentExecutor with OPENAI_FUNCTIONS

manoharnv commented 6 months ago

System Info

Langchain version 0.352

SystemMessage is ignored when I invoke AgentExecutor.run function. the code looks as below.

from typing import Tuple, Dict

from langchain.agents import initialize_agent, AgentType
from langchain.agents.agent import AgentExecutor
from langchain.agents.format_scratchpad.openai_functions import format_to_openai_function_messages
from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
from langchain.memory import ConversationBufferMemory
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.tools.render import format_tool_to_openai_function
from langchain_core.messages import SystemMessage

from elasticsearch_agent.config import cfg
from elasticsearch_agent.tools.index_data_tool import IndexShowDataTool
from elasticsearch_agent.tools.index_details_tool import IndexDetailsTool
from elasticsearch_agent.tools.index_search_tool import create_search_tool
from elasticsearch_agent.tools.list_indices_tool import ListIndicesTool

tools = [
    ListIndicesTool(),
    IndexShowDataTool(),
    IndexDetailsTool(),
    create_search_tool(),
]

def elastic_agent_factory() -> AgentExecutor:
    system_msg = """
        You are a helpful AI ElasticSearch Expert Assistant
        **Always you will get the field names of the ElasticSearch index from the Elasticsearch DB as a first step.
        You are provided with various tools to help the user to get information from an ElasticSearch index.
        you will get the index name from the question. If not provided, show the list of available indices and  ask the user to choose it.
        You will generate required aggregation queries for any analytical questions asked.
        You will use 'aggregations' field in response object for answering analytical queries.
        Dont's:
         Never assume index names or field names.
        """

    agent_kwargs, memory = setup_memory()
    agent_kwargs["system_message"] = SystemMessage(content=system_msg)

    return initialize_agent(
        tools,
        cfg.llm,
        agent=AgentType.OPENAI_FUNCTIONS,
        verbose=False,
        agent_kwargs=agent_kwargs,
        memory=memory
    )

def setup_memory() -> Tuple[Dict, ConversationBufferMemory]:
    """
    Sets up memory for the open ai functions agent.
    :return a tuple with the agent keyword pairs and the conversation memory.
    """
    agent_kwargs = {
        "extra_prompt_messages": [MessagesPlaceholder(variable_name="memory")],
    }
    memory = ConversationBufferMemory(memory_key="memory", return_messages=True)

    return agent_kwargs, memory

if __name__ == "__main__":
    agent_executor = elastic_agent_factory()
    prompt = agent_executor.agent.prompt
    print(prompt)
    print(type(agent_executor.agent.prompt))

Who can help?

No response

Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[X] Chains
[ ] Callbacks/Tracing
[X] Async

Reproduction

run the given code

Expected behavior

The chain agent should consider using the system message and extra prompt message provided to it.

dosubot[bot] commented 6 months ago

🤖

Hello @manoharnv! I'm Dosu, your friendly bot helper. I'm here to assist you in resolving bugs, answering questions, and aiding in your contribution journey. While we wait for a human maintainer, I'll do my best to provide you with the support you need. Looking forward to our interaction!

Based on the information you provided and the context from the LangChain repository, it seems you want to instantiate an OpenAI Functions Agent with both memory and a custom system message.

To achieve this, you can include the system message in the agent_kwargs when initializing the agent. Here's how you can do it:

system_message = SystemMessage(content="You are Jelly.")

agent_kwargs = {
    "extra_prompt_messages": [MessagesPlaceholder(variable_name="memory")],
    "system_message": system_message,
}

open_ai_agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.OPENAI_FUNCTIONS,
    verbose=True,
    agent_kwargs=agent_kwargs,
    memory=memory,
)

This solution was confirmed to work in LangChain version 0.0.234. However, it was noted that the system_message argument was discontinued in later versions, and the system_message should be passed as a string within agent_kwargs. You can refer to this issue for more details.

If you want to use the from_llm_and_tools method of the OpenAIMultiFunctionsAgent class, you can do it like this:

from langchain.chains import LLMMathChain
from langchain.llms import OpenAI
from langchain.utilities import SerpAPIWrapper, SQLDatabase
from langchain_experimental.sql import SQLDatabaseChain
from langchain.agents import Tool, AgentType
from langchain.agents.openai_functions_multi_agent.base import OpenAIMultiFunctionsAgent
from langchain.chat_models import ChatOpenAI
from langchain.prompts import MessagesPlaceholder
from langchain.memory import ConversationBufferMemory

# Assuming you have a language model and tools
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
search = SerpAPIWrapper()
llm_math_chain = LLMMathChain.from_llm(llm=llm, verbose=True)
db = SQLDatabase.from_uri("sqlite:///../../../../../notebooks/Chinook.db")
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)
tools = [
    Tool(
        name="Search",
        func=search.run,
        description="useful for when you need to answer questions about current events. You should ask targeted questions",
    ),
    Tool(
        name="Calculator",
        func=llm_math_chain.run,
        description="useful for when you need to answer questions about math",
    ),
    Tool(
        name="FooBar-DB",
        func=db_chain.run,
        description="useful for when you need to answer questions about FooBar. Input should be in the form of a question containing full context",
    ),
]

# Create a custom system message
custom_system_message = MessagesPlaceholder(variable_name="memory")

# Create the memory
memory = ConversationBufferMemory(memory_key="memory", return_messages=True)

# Create the agent
agent = OpenAIMultiFunctionsAgent.from_llm_and_tools(
    llm,
    tools,
    extra_prompt_messages=[custom_system_message],
)

In this example, llm is an instance of ChatOpenAI which is the language model to use. tools is a list of tools the agent has access to. memory is the memory instance that allows the agent to remember intermediate steps. extra_prompt_messages is the custom system message to use. If not provided, a default one will be used.

I hope this helps! If you have any other questions, feel free to ask.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot. Sources:

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

manoharnv commented 6 months ago

@dosu-bot as you see, I am already doing the same as you suggested. but looks like system messages are not coming into effect.

dosubot[bot] commented 6 months ago

🤖

I'd be happy to help you with that. Could you please provide me with more details about the issue you're experiencing? Specifically, what system messages are you referring to, and how are you expecting them to come into effect?

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

manoharnv commented 6 months ago

@dosu-bot here is the code.

when I invoke agentExecutor, it is supposed to first fetch the field details, map the fields for the given question and then run the aggregation query. but it is directly trying to query the elk with its own field names. agent invocation code:

agentExecutor.run("""get top 3 department, occurrences and top 3 users in each department in the past 30 days
index=kibana_telemetry""")

agent code:

def elastic_agent_factory() -> AgentExecutor:
    system_msg = """
        You are a helpful AI ElasticSearch Expert Assistant
        **Always you will get the field names of the ElasticSearch index from the Elasticsearch DB as a first step.
        You are provided with various tools to help the user to get information from an ElasticSearch index.
        you will get the index name from the question. If not provided, show the list of available indices and  ask the user to choose it.
        You will generate required aggregation queries for any analytical questions asked.
        You will use 'aggregations' field in response object for answering analytical queries.
        Dont's:
         Never assume index names or field names.
        """

    agent_kwargs, memory = setup_memory()
    agent_kwargs["system_message"] = SystemMessage(content=system_msg)

    return initialize_agent(
        tools,
        cfg.llm,
        agent=AgentType.OPENAI_FUNCTIONS,
        verbose=False,
        agent_kwargs=agent_kwargs,
        memory=memory
    )

dosubot[bot] commented 6 months ago

🤖

Based on the context provided, there was a similar issue that was resolved in the past. The solution was to include the system message in the agent_kwargs when initializing the agent. However, it was noted that the system_message argument was discontinued in later versions, and the system_message should be passed as a string within agent_kwargs. Here is the reference to the issue.

However, for AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION, the system message is not directly customizable through agent_kwargs. Instead, you can modify the PREFIX, FORMAT_INSTRUCTIONS, and SUFFIX variables in the prompt.py file. These variables contain the default system messages.

Alternatively, you can also replace the default system messages when creating an instance of StructuredChatAgent as follows:

agent = StructuredChatAgent.from_llm_and_tools(
    llm=your_llm,
    tools=your_tools,
    prefix="Your custom prefix here.",
    suffix="Your custom suffix here.",
    format_instructions="Your custom instructions here."
)

Please replace "Your custom prefix here.", "Your custom instructions here.", and "Your custom suffix here." with your actual custom messages.

You can find more details in the source code here: prompt.py and base.py.

To ensure that the agent fetches the field details from the Elasticsearch database before running the query when you invoke the AgentExecutor.run function, you need to add a method in the ElasticsearchChatMessageHistory class that fetches the field details from the Elasticsearch database.

You can add a method like get_field_details to fetch the field details:

def get_field_details(self, field_name: str) -> Any:
    """Fetch the details of a specific field from Elasticsearch"""
    try:
        from elasticsearch import ApiError

        result = self.client.get_field_mapping(
            index=self.index,
            fields=[field_name],
        )
    except ApiError as err:
        logger.error(f"Could not retrieve field details from Elasticsearch: {err}")
        raise err

    if result and self.index in result and 'mappings' in result[self.index] and field_name in result[self.index]['mappings']:
        return result[self.index]['mappings'][field_name]
    else:
        return None

Then, in your AgentExecutor.run function, you can call this method before running the query to ensure that the system message is considered and the field details are fetched:

def run(self, agent, system_message):
    # Fetch field details from Elasticsearch
    field_details = agent.memory.get_field_details('system_message')
    # Consider the system message and field details before running the query
    # ...

Please note that you need to replace 'system_message' with the actual field name you want to fetch from Elasticsearch. Also, the run function is not provided in the context, so you need to adjust the code according to your actual run function implementation.

You can find more details in the source code here: elasticsearch.py.

Unfortunately, I wasn't able to find the exact structure and functionality of the AgentExecutor.run function in the LangChain codebase. It would be helpful if you could provide more details about the run function or any error messages you're encountering.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

langchain-ai / langchain