Unable to use non-Gemini models through ChatVertexAI

jarib commented 2 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code


import { ChatVertexAI } from '@langchain/google-vertexai'

const chat = new ChatVertexAI({
    model: 'claude-3-opus@20240229',
    temperature: 0.0,
    maxOutputTokens: 1200,
})

const result = await chat.invoke('Tell me a story')

console.log(result)

Error Message and Stack Trace (if applicable)

Error: Unable to verify model params: {"lc":1,"type":"constructor","id":["langchain","chat_models","chat_integration","ChatVertexAI"],"kwargs":{"model":"claude-3-opus","temperature":0,"max_output_tokens":1200,"platform_type":"gcp"}}
    at validateModelParams (file:///Users/redacted/src/langchain-repro/node_modules/@langchain/google-common/dist/utils/common.js:100:19)
    at copyAndValidateModelParamsInto (file:///Users/redacted/src/langchain-repro/node_modules/@langchain/google-common/dist/utils/common.js:105:5)
    at new ChatGoogleBase (file:///Users/redacted/src/langchain-repro/node_modules/@langchain/google-common/dist/chat_models.js:191:9)
    at new ChatGoogle (file:///Users/redacted/src/langchain-repro/node_modules/@langchain/google-gauth/dist/chat_models.js:12:9)
    at new ChatVertexAI (file:///Users/redacted/src/langchain-repro/node_modules/@langchain/google-vertexai/dist/chat_models.js:10:9)
    at main (file:///Users/redacted/src/langchain-repro/test.js:5:22)
    at file:///Users/redacted/src/langchain-repro/test.js:22:1
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)

Description

I am trying to use ChatVertexAI with Anthropic Claude 3, but it seems this class only supports Gemini models and returns the above error message.

This appears to be a deliberate choice in the code:

https://github.com/langchain-ai/langchainjs/blob/b9d86b16bd0788144f8f27cf5d18960c43da49af/libs/langchain-google-common/src/utils/common.ts#L125-L132

I've verified that using Claude and Vertex AI and Anthropics Vertex SDK works fine:

import { AnthropicVertex } from '@anthropic-ai/vertex-sdk'

const projectId = 'my-project-id'
const region = 'us-east5'

// Goes through the standard `google-auth-library` flow.
const client = new AnthropicVertex({
    projectId,
    region,
})

async function main() {
    const result = await client.messages.create({
        model: 'claude-3-opus@20240229',
        max_tokens: 100,
        messages: [
            {
                role: 'user',
                content: 'Hey Claude!',
            },
        ],
    })
    console.log(JSON.stringify(result, null, 2))
}

main()

Output:

{
  "id": "msg_vrtx_01M1yGR5LiteHznRmyK2MaPG",
  "type": "message",
  "role": "assistant",
  "model": "claude-3-opus-20240229",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 10,
    "output_tokens": 12
  },
  "content": [
    {
      "type": "text",
      "text": "Hello! How can I assist you today?"
    }
  ],
  "stop_reason": "end_turn"
}

System Info

    "@anthropic-ai/vertex-sdk": "^0.3.5",
    "@google-cloud/vertexai": "^1.1.0",
    "@langchain/google-vertexai": "^0.0.11",
    "langchain": "^0.1.35"

Platform: Mac Node: 20.11.0

dosubot[bot] commented 2 months ago

To address the issue of using Anthropic Claude 3 with LangChain.js, particularly with the ChatVertexAI class, it's important to note that the current implementation indeed focuses on Gemini models, leading to the error you encountered. However, you can integrate Claude 3 by adapting the approach used for Anthropic models in LangChain.js, specifically through a custom implementation similar to the ChatAnthropic class.

Here's a concise guide to get you started:

Ensure Dependencies: Make sure you have both LangChain and the Anthropic SDK installed in your project.
Use the ChatAnthropic Class: Import and configure the ChatAnthropic class from LangChain to work with Claude 3. You'll need to specify the model name, API key, and other relevant settings like temperature and max tokens.
Invoke the Model: Utilize the invoke method for sending prompts to Claude 3 and receiving responses. For interactive applications, consider exploring streaming responses.
Customize as Needed: Tailor the integration to fit your application's needs, adjusting prompt formatting, response handling, or integrating with other services as required.

This approach allows you to leverage Claude 3's capabilities within LangChain.js applications. For detailed implementation and examples, refer to the ChatAnthropic class in the LangChain.js repository and the associated documentation.

Keep in mind, this solution circumvents the ChatVertexAI class's current limitations with non-Gemini models. The LangChain.js team is working on expanding support for a broader range of models, so staying updated with the repository may provide more streamlined solutions in the future.

#### Details

**Sources** - [Integrating langchainJS with fine tuned gemini model](https://github.com/langchain-ai/langchainjs/discussions/5153) - [Plugins support in js](https://github.com/langchain-ai/langchainjs/issues/1157) - [libs/langchain-anthropic/src/chat_models.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-anthropic/src/chat_models.ts) - [examples/src/models/chat/integration_anthropic.ts](https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/integration_anthropic.ts) - [docs/core_docs/docs/integrations/chat/anthropic.mdx](https://github.com/langchain-ai/langchainjs/blob/main/docs/core_docs/docs/integrations/chat/anthropic.mdx) - [libs/langchain-anthropic/src/tests/chat_models.int.test.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-anthropic/src/tests/chat_models.int.test.ts) - [langchain/src/chat_models/anthropic.ts](https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/chat_models/anthropic.ts) - [libs/langchain-community/src/chat_models/googlevertexai/web.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/src/chat_models/googlevertexai/web.ts) - [libs/langchain-google-vertexai/src/chat_models.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-vertexai/src/chat_models.ts) - [examples/src/models/chat/integration_googlevertexai.ts](https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/integration_googlevertexai.ts) - [libs/langchain-google-common/src/utils/common.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-common/src/utils/common.ts) - [examples/src/models/chat/token_usage_tracking_anthropic.ts](https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/token_usage_tracking_anthropic.ts) - [examples/src/models/chat/integration_anthropic_legacy.ts](https://github.com/langchain-ai/langchainjs/blob/main/examples/src/models/chat/integration_anthropic_legacy.ts) - [libs/langchain-anthropic/README.md](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-anthropic/README.md) - [libs/langchain-anthropic/package.json](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-anthropic/package.json)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

afirstenberg commented 2 months ago

Similar to #2562

The issue is that Claude in Vertex AI is neither the same protocol as Gemini, nor quite the same as what is offered through Anthropic. So we need something that combines the two.

Do-able, but hasn't risen in priority.

My general thoughts about how to do this would be

The Anthropic library (and possibly the the @langchain/anthropic library) would be a build dependency and optional dependency to use it. Probably use this for the types, mostly, and parsing the results.
Add a utility class for anthropic.
- The things that call baseMessageToContent, safeResponseToChatGeneration, and safeResponseToChatResult need to call their equivalents in this class
Changes in GoogleAIConnection
- modelFamily() should add "claude" or something along those lines. And GoogleLLMModelFamily should be adjusted as well (probably).
- buildUrlVertex() needs a change to handle non-Google models (possibly determine that based on the model family?)
Probably make a subclass of GoogleAIConnection or AbstractGoogleLLMConnection to handle the Claude layout. But I haven't dug into the details.

The good news is that the authentication part is handled by default with the Connection classes.

ianwoodfill commented 4 weeks ago

+1, would be a very useful capability. Seems like it's already supported in BedrockChat, which is great.

yermie commented 2 weeks ago

+1, especially given the renewed interest in Claude 3.5 Sonnet

DanKupisz commented 1 week ago

+1 for Claude 3.5 Sonnet

jeloi commented 1 week ago

+1 another vote - Claude 3.5 support through Vertex AI is a key integration for companies on GCP to adopt Langchain.

afirstenberg commented 1 week ago

I see the +1s - this is now next on my priority list!

I'm aiming to get #5835 into a stable state ASAP so it can be merged, since it contains some updates that would be useful for this effort as well. The approach I outlined above still mostly holds.

Sorry I couldn't get this in place for I/O Connect, but hope to have good news soon!.

MJCyto commented 4 days ago

Could you extend to other LLMs in the model garden such as Llama 3?

afirstenberg commented 4 days ago

Yes... but...

In the Vertex AI Model Garden, Llama 3 requires you to deploy the model to an endpoint you control. And the details of the API are poorly documented. Claude has more full API support with "model-as-a-service" and a standard endpoint.

My current thinking is that I'll be able to provide direct support through the current classes we have (ie - you'll just be able to specify a claude model and it will work), while you may need to do some work for other models in the Model Garden, including deploying an endpoint.

langchain-ai / langchainjs