Error When Using Chat Input Node with Image File in Flow

[adielgur]

Bug Description

I'm trying to test a flow that utilizes GPT-4o to analyze an image, specifically by using the Chat Input node with an image file. However, I’m encountering the following error when trying to pass the image:

Error building Component Chat Input:

(builtins.TypeError) Object of type Image is not JSON serializable [SQL: INSERT INTO message (timestamp, sender, sender_name, session_id, text, id, flow_id, files) VALUES (%(timestamp)s, %(sender)s, %(sender_name)s, %(session_id)s, %(text)s, %(id)s::UUID, %(flow_id)s::UUID, %(files)s)] [parameters: [{'id': UUID('4b2ccb05-19f6-4f90-9bd8-51a3c6b4c411'), 'flow_id': '7e2ed8c2-47ea-4961-ad3b-b750f7872e6d', 'sender': 'User', 'files': [Image(path='/app/d ... )], 'session_id': '7e2ed8c2-47ea-4961-ad3b-b750f7872e6d', 'sender_name': 'User', 'timestamp': datetime.datetime(2024, 9, 17, 17:32:32), 'text': ''}]]

### Reproduction

1. Set up a flow to analyze images using the Chat Input node.
2. Attempt to pass an image file through the Chat Input node.
3. The error occurs during this process, preventing the image analysis flow from proceeding.

### Expected behavior

Hit "Play" when attaching a file, without an error

### Who can help?

_No response_

### Operating System

Docker / Local python installation both don't work

### Langflow Version

latest

### Python Version

3.12

### Screenshot

![chatinput](https://github.com/user-attachments/assets/13b87986-e829-415b-897f-8fc5fd9c4c14)

### Flow File

_No response_

[dosubot[bot]]

Hey @adielgur! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

The error you're encountering, Object of type Image is not JSON serializable, indicates that the image file is not being properly serialized before being stored in the database. To resolve this, you need to ensure that the image is converted to a JSON-serializable format, such as a base64-encoded string.

Langflow provides utility methods for handling this conversion. Here’s how you can address the issue:

1. Convert the Image to Base64: Use the pil_to_base64 function to convert the image to a base64-encoded string.

   from PIL import Image as PILImage
   from io import BytesIO
   import base64
   
   def pil_to_base64(image: PILImage) -> str:
      buffered = BytesIO()
      image.save(buffered, format="PNG")
      img_str = base64.b64encode(buffered.getvalue())
      return img_str.decode("utf-8")

2. Update the ChatInput Node: Modify the ChatInput node to convert the image file to a base64 string before storing it.

   class ChatInput(ChatComponent):
      # ... other code ...
   
      def message_response(self) -> Message:
          # Convert image files to base64
          base64_files = []
          for file in self.files:
              if isinstance(file, Image):
                  base64_files.append(pil_to_base64(file))
              else:
                  base64_files.append(file)
   
          message = Message(
              text=self.input_value,
              sender=self.sender,
              sender_name=self.sender_name,
              session_id=self.session_id,
              files=base64_files,
          )
   
          if (
              self.session_id
              and isinstance(message, Message)
              and isinstance(message.text, str)
              and self.should_store_message
          ):
              store_message(
                  message,
                  flow_id=self.graph.flow_id,
              )
              self.message.value = message
   
          self.status = message
          return message

3. Ensure Proper Serialization: Make sure the Message class and any related serialization logic handle the base64-encoded string correctly.

By converting the image to a base64 string, you ensure that it is JSON-serializable and can be stored in the database without issues. This should resolve the error you're encountering when using the Chat Input node with an image file ^[1][2].

To continue talking to Dosu, mention @dosu.

[adielgur]

Doesn't work, I also tried to uncheck "store messages" and getting a different error still.

[h-arnold]

I've experienced the same bug via the Chat Input component in the UI and when passing a file (.png image files) via the API.

The issue is the same running on a vanilla x86 docker instance and on Google Cloud run.

Interestingly, passing an image in the playground does work so I'm guessing there must be a difference between how images are handled in the playground chat interface than the Chat Input component or the API.

I've had the issue on 1.017 but haven't tested it on 1.018 yet.

[Cristhianzl]

on it