VertexAIImageGeneratorChat not compatible with chain

[axnsantana]

I am replicating the code in langchain docs but using a chain.

Code to replicate error

from langchain_google_vertexai.vision_models import VertexAIImageGeneratorChat
from langchain_core.prompts import PromptTemplate

model = VertexAIImageGeneratorChat()

prompt = PromptTemplate(
    template="I want an image of {img_object} in {img_context}.",
    input_variables=["img_object", "img_context"],
)

chain = prompt | model

response = chain.invoke(dict(img_object="cat", img_context="beach"))

print(response)

Traceback

Traceback (most recent call last):
  File "/Users/alex.santana/Library/Application Support/JetBrains/PyCharm2024.1/scratches/scratch_18.py", line 15, in <module>
    response = chain.invoke(dict(img_object="cat", img_context="beach"))
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2824, in invoke
    input = step.invoke(input, config)
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 265, in invoke
    self.generate_prompt(
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 698, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 555, in generate
    raise e
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 545, in generate
    self._generate_with_cache(
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 770, in _generate_with_cache
    result = self._generate(
  File "/Users/alex.santana/Projects/***************************/venv/lib/python3.9/site-packages/langchain_google_vertexai/vision_models.py", line 466, in _generate
    raise ValueError(
ValueError: Only one message with one text part allowed for image generation Must The prompt of the image

BUG

The bug is in a validation point in this code

is_valid = len(messages) == 1 and len(messages[0].content) == 1
if is_valid:
    user_query = get_text_str_from_content_part(messages[0].content[0])
if user_query is None:
    raise ValueError(
        "Only one message with one text part allowed for image generation"
        " Must The prompt of the image"
    )

In the chain, the message content is a string and not a list as in the example from langchain docs. Thus, any usable prompt string will have length greater more than 1

Library versions

langchain==0.1.19
langchain-community==0.0.38
langchain-core==0.2.21
langchain-google-genai==1.0.3
langchain-google-vertexai==1.0.6
langchain-openai==0.1.6
langchain-text-splitters==0.0.1

platform: macOS 12.7 python version: 3.9.16

[lkuligin]

@jzaldi

[jzaldi]

We were expecting a multipart message, but it makes sense to support string context for this model so I'll add the support in #393.

In the meanwhile this snippet works:

from langchain_google_vertexai.vision_models import VertexAIImageGeneratorChat
from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate

model = VertexAIImageGeneratorChat()

prompt = ChatPromptTemplate.from_messages(
    [
        HumanMessagePromptTemplate.from_template(template=[
            {"type": "text", "text": "I want an image of {img_object} in {img_context}."}
        ])
    ]
)

chain = prompt | model

response = chain.invoke(dict(img_object="cat", img_context="beach"))

print(response)