SystemMessage Not Effective

babelpainterwell commented 2 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.

Example Code

system_prompt = "Don't speak antyhing."
agent = AgentM(model, tools)

# the initial task input
# messages = [HumanMessage(content="What is score of NBA Final Game 1?")]
messages = [HumanMessage(content="The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? ")]
messages = [SystemMessage(content=system_prompt)] + messages

# Execute the task
config = {"configurable": {"thread_id": "1"}}
try: 
  result = agent.graph.invoke({"messages": messages}, config=config)
  print(result['messages'][-1].content)
  print("*" * 20)
  print(agent.graph.get_state(config))
except UserExitError as e:
  print(e.message)

Error Message and Stack Trace (if applicable)

****% python tutorial.py Please provide your student ID so I can assist you in calculating the total number of students in your class.

StateSnapshot(values={'messages': [SystemMessage(content="Don't say antyhing.", id='ffc8061c-796f-430e-9d7f-9c82852e4a12'), HumanMessage(content='The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? ', id='a1e0d963-eda8-46f4-b3c3-3ae26780c08b'), AIMessage(content='Please provide your student ID so I can assist you in calculating the total number of students in your class.', response_metadata={'token_usage': {'completion_tokens': 22, 'prompt_tokens': 150, 'total_tokens': 172}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-f5cdb365-3cc9-4025-80f0-0d80fd2d6d34-0', usage_metadata={'input_tokens': 150, 'output_tokens': 22, 'total_tokens': 172})], 'query_human': False}, next=(), config={'configurable': {'thread_id': '1', 'thread_ts': '1ef26ada-893a-6892-8001-0bd642095671'}}, metadata={'source': 'loop', 'step': 1, 'writes': {'llm': {'messages': [AIMessage(content='Please provide your student ID so I can assist you in calculating the total number of students in your class.', response_metadata={'token_usage': {'completion_tokens': 22, 'prompt_tokens': 150, 'total_tokens': 172}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-f5cdb365-3cc9-4025-80f0-0d80fd2d6d34-0', usage_metadata={'input_tokens': 150, 'output_tokens': 22, 'total_tokens': 172})], 'query_human': False}}}, created_at='2024-06-09T22:14:37.122152+00:00', parent_config={'configurable': {'thread_id': '1', 'thread_ts': '1ef26ada-80c4-6d52-8000-dd00aa835319'}})

Description

SystemMessage can be correctly appended in the beginning, by printing out graph.get_state(), but it seems the agent can't be guided by any info in the SystemMessage.

System Info

System Information

OS: Darwin OS Version: Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 Python Version: 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:37:23) [Clang 13.0.0 (clang-1300.0.29.30)]

Package Information

langchain_core: 0.2.4 langchain: 0.2.2 langchain_community: 0.2.3 langsmith: 0.1.74 langchain_anthropic: 0.1.15 langchain_openai: 0.1.8 langchain_text_splitters: 0.2.1 langgraph: 0.0.64

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

hinthornw commented 2 months ago

Hi @babelpainterwell - thanks for raising this issue. Do you have a code snippet of what your actual graph looks like? Hard to give specific advice without it.

It seems from what you've shared that you're trying to instruct the LLM to never generate a response using the system message. Depending on the model, it's sometimes quite hard to get them actually to not respond (i.e., immediately generate a stop token) - it's sometimes easier to have them generate a canned "end" phrase before it like "NO RESPONSE" or something, though even that is imperfect.

IIRC, this is more of a prompt engineering / machine learning problem that you're running into rather than an issue with the actual LangGraph framework. With these problems, I typically recommend writing tests and then modifying the prompt to pass these tests (i.e., eval-driven development). LangSmith is one such tool that makes this quite convenient.

hinthornw commented 2 months ago

Gonna move this to discussions since at first glance it appears there isn't an actual bug with LangGraph associated with this ticket. Open to being convinced otherwise (can re-open if so)

langchain-ai / langgraph