ChatPromtpTemplate doesn't accept PDF data as bytes

alandema commented 1 day ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

pdf_data = base64.b64encode(httpx.get("https://dagrs.berkeley.edu/sites/default/files/2020-01/sample.pdf").content).decode("utf-8")

prompt = ChatPromptTemplate([
    ("system", "You are a helpful assistant "),
    ("human", [
        {"type": "media", "mime_type": "application/pdf", "data": pdf_data},
        {"type": "text", "text": "{user_input}"}
    ])
])

Error Message and Stack Trace (if applicable)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], [line 12](vscode-notebook-cell:?execution_count=4&line=12)
      [1](vscode-notebook-cell:?execution_count=4&line=1) model = ChatGoogleGenerativeAI(
      [2](vscode-notebook-cell:?execution_count=4&line=2)     model="gemini-1.5-flash",
      [3](vscode-notebook-cell:?execution_count=4&line=3)     temperature=0.1,
   (...)
      [6](vscode-notebook-cell:?execution_count=4&line=6)     max_retries=2
      [7](vscode-notebook-cell:?execution_count=4&line=7) )
      [9](vscode-notebook-cell:?execution_count=4&line=9) pdf_data = base64.b64encode(
     [10](vscode-notebook-cell:?execution_count=4&line=10)     httpx.get("http://www.dagrs.berkley.edu/sites/default/files/2020-01/sample.pdf").content).decode("utf-8")
---> [12](vscode-notebook-cell:?execution_count=4&line=12) prompt = ChatPromptTemplate([
     [13](vscode-notebook-cell:?execution_count=4&line=13)     ("system", "You are a helpful assistant "),
     [14](vscode-notebook-cell:?execution_count=4&line=14)     ("human", [
     [15](vscode-notebook-cell:?execution_count=4&line=15)         {"type": "media", "mime_type": "application/pdf", "data": pdf_data},
     [16](vscode-notebook-cell:?execution_count=4&line=16)         {"type": "text", "text": "{user_input}"}
     [17](vscode-notebook-cell:?execution_count=4&line=17)     ])
     [18](vscode-notebook-cell:?execution_count=4&line=18) ])

File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:992, in ChatPromptTemplate.__init__(self, messages, template_format, **kwargs)
    [938](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:938) def __init__(
    [939](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:939)     self,
    [940](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:940)     messages: Sequence[MessageLikeRepresentation],
   (...)
    [943](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:943)     **kwargs: Any,
    [944](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:944) ) -> None:
    [945](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:945)     """Create a chat prompt template from a variety of message formats.
    [946](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:946) 
    [947](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:947)     Args:
   (...)
    [990](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:990) 
    [991](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:991)     """
--> [992](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:992)     _messages = [
    [993](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:993)         _convert_to_message(message, template_format) for message in messages
    [994](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:994)     ]
    [996](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:996)     # Automatically infer input variables from messages
    [997](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:997)     input_vars: set[str] = set()

File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:993, in <listcomp>(.0)
    [938](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:938) def __init__(
    [939](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:939)     self,
    [940](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:940)     messages: Sequence[MessageLikeRepresentation],
   (...)
    [943](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:943)     **kwargs: Any,
    [944](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:944) ) -> None:
    [945](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:945)     """Create a chat prompt template from a variety of message formats.
    [946](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:946) 
    [947](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:947)     Args:
   (...)
    [990](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:990) 
    [991](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:991)     """
    [992](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:992)     _messages = [
--> [993](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:993)         _convert_to_message(message, template_format) for message in messages
    [994](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:994)     ]
    [996](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:996)     # Automatically infer input variables from messages
    [997](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:997)     input_vars: set[str] = set()

File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:1454, in _convert_to_message(message, template_format)
   [1452](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1452) message_type_str, template = message
   [1453](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1453) if isinstance(message_type_str, str):
-> [1454](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1454)     _message = _create_template_from_message_type(
   [1455](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1455)         message_type_str, template, template_format=template_format
   [1456](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1456)     )
   [1457](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1457) else:
   [1458](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1458)     _message = message_type_str(
   [1459](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1459)         prompt=PromptTemplate.from_template(
   [1460](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1460)             cast(str, template), template_format=template_format
   [1461](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1461)         )
   [1462](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1462)     )

File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:1365, in _create_template_from_message_type(message_type, template, template_format)
   [1351](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1351) """Create a message prompt template from a message type and template string.
   [1352](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1352) 
   [1353](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1353) Args:
   (...)
   [1362](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1362)     ValueError: If unexpected message type.
   [1363](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1363) """
   [1364](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1364) if message_type in ("human", "user"):
-> [1365](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1365)     message: BaseMessagePromptTemplate = HumanMessagePromptTemplate.from_template(
   [1366](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1366)         template, template_format=template_format
   [1367](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1367)     )
   [1368](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1368) elif message_type in ("ai", "assistant"):
   [1369](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1369)     message = AIMessagePromptTemplate.from_template(
   [1370](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1370)         cast(str, template), template_format=template_format
   [1371](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1371)     )

File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:565, in _StringImageMessagePromptTemplate.from_template(cls, template, template_format, partial_variables, **kwargs)
    [563](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:563)             prompt.append(img_template_obj)
    [564](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:564)         else:
--> [565](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:565)             raise ValueError(f"Invalid template: {tmpl}")
    [566](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:566)     return cls(prompt=prompt, **kwargs)
    [567](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:567) else:

ValueError: Invalid template: {'type': 'media', 'mime_type': 'application/pdf', 'data':

Description

I'm trying to send a PDF file as bytes to Gemini using the ChatPromptTemplate method.

Sending the PDF to Gemini seems to works fine as per #4589 and #215

But when using the ChatPromptTemplate, it appears that the code here is not ready to deal with PDFs type in the prompt template.

System Info

langchain==0.3.2 langchain-community==0.3.1 langchain-core==0.3.9 langchain-google-genai==2.0.0 langchain-text-splitters==0.3.0 langgraph==0.2.34 langgraph-checkpoint==2.0.0 langsmith==0.1.131

MadisonMajarais commented 17 hours ago

Hi, we're a group of students from the University of Toronto, and we're interested in investigating this issue further and making a PR to fix it. Any additional information about this issue is appreciated! Thanks!

alandema commented 15 hours ago

Hi, we're a group of students from the University of Toronto, and we're interested in investigating this issue further and making a PR to fix it. Any additional information about this issue is appreciated! Thanks!

Hi.

I've tracked down the problem to that specific section of the code, where an "elif" is missing for "data" types. It should have something like:

elif isinstance(tmpl, dict) and "data" in tmpl:
    #TODO

The problem seems to be really in the ChatPromptTemplate construction, because if you rewrite the example as follows, you can create prompt templates with PDF as bytes without errors.

prompt_msgs = [
    SystemMessage(
        content="You're a helpful AI assistant"
    ),
    HumanMessage(content=[{"type": "media", "mime_type": "application/pdf", "data": pdf_data}]
                 ),
    HumanMessagePromptTemplate.from_template(
        "Reply this query: {query}"
    ),
]

prompt = ChatPromptTemplate.from_messages(messages=prompt_msgs)

chain = prompt | model

response = chain.invoke({"query": "Is the sky blue?"})

langchain-ai / langchain