ChatGoogleGenerativeAI sends `SystemMessage` as `HumanMessage` instead of setting the parameters on the model itself.

langchain-ai / langchainjs

🦜🔗 Build context-aware reasoning applications 🦜🔗

MIT License

12.46k stars 2.11k forks source link

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

It happens right here. https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L62-L64

And the broken tests too https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/tests/chat_models.test.ts#L234-L237

Error Message and Stack Trace (if applicable)

No response

Description

As the title says, the system message is sent as the user instead of being set to the model via the systemInstruction param.

There are length limits on the messages that can be sent, so if the SystemMessage is too long the request will be blocked, but if the param is set properly the length restriction goes away.

Either expose the systemInstruction param on the constructor or auto set it when the first message is a SystemMessage and remove it from the contents being sent.

Monkey patched by separating the two roles: https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L62-L64

Removed the marge https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L225-L227

And did what I proposed earlier https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/chat_models.ts#L783-L791

System Info

Windows 11, node v22.x, yarn 1.x

I found a similar closed issue, #2095, which was last updated on August 07, 2023. However, it has no comments or solution provided ^[1].

To address the issue where SystemMessage is incorrectly sent as a user message instead of being set via the systemInstruction parameter, you should ensure that the systemMessageToContent function correctly uses the useSystemInstruction flag to determine the role of the message. The current implementation checks this flag:

async function systemMessageToContent(
  message: SystemMessage,
  useSystemInstruction: boolean
): Promise<GeminiContent[]> {
  return useSystemInstruction
    ? roleMessageToContent("system", message)
    : [
        ...(await roleMessageToContent("user", message)),
        ...(await roleMessageToContent("model", new AIMessage("Ok"))),
      ];
}

Make sure that when calling systemMessageToContent, the useSystemInstruction parameter is set to true to avoid the message being sent as a user message. If this parameter is not being set correctly elsewhere in the code, adjust the logic where systemMessageToContent is invoked to pass the correct value for useSystemInstruction ^[2].

Additionally, ensure that the systemInstruction parameter is properly set and used within the formatData method of the AbstractGoogleLLMConnection class. This method prepares a GeminiRequest object, which includes the systemInstruction if it has a defined role and non-empty parts ^[3]^[4].

To continue talking to Dosu, mention @dosu.

langchain-ai / langchainjs