Add Image Upload to Chat to allow for creating code from screenshots

[kleneway]

Add Image Upload Feature to Chat Interface

Description:

Add a new button to the chat interface that allows users to upload an image. The image will be sent to the /api/image/upload API endpoint, and the returned URL will be saved and passed along with future chat messages to the backend. The uploaded images should be displayed in the chat interface immediately after upload and should be passed to the chat message api.

Requirements:

• Upload Button:

  • Add an upload button with an upload icon to the ChatInput component.
  • Place the upload button to the left of the "send message" button (the up arrow).
  • Style the button consistently with other buttons in the chat interface using Tailwind CSS.

• File Picker Dialog:

  • Open a default file picker dialog when the upload button is clicked.
  • Allow users to select multiple images at once.

• Image Validation:

  • Validate the image before uploading:
  • Must be under 20MB.
  • Must be either PNG or JPEG.
  • Show an error message using a toast notification if a non-image file is uploaded.
  • Display an error message using a toast notification if the image exceeds 20MB.

• Image Upload:

  • Send the selected image(s) to the /api/image/upload API endpoint.
  • Handle multiple image uploads in parallel.
  • Set a loading state to true during the upload and back to false once the upload is complete.
  • Display a loading indicator (e.g., spinner) on the upload button during the upload process.

• API Interaction:

  • Use the existing API route src/app/api/image/upload/route.ts for handling image uploads.
  • Ensure the API interaction follows the existing patterns and conventions in the codebase.

• Error Handling:

  • Display an error message using a toast notification if the upload fails.
  • Handle API errors gracefully and provide user-friendly error messages.

• Display Uploaded Images:

  • Save the returned URL from the API and pass it along with future chat messages.
  • Display the uploaded images in the chat interface immediately after upload.
  • Ensure the images are displayed correctly in the chat bubbles, maintaining the existing chat message styles.

• State Management:

  • Use useState to manage the loading state and image URLs.
  • Ensure the loading state is managed correctly during the upload process.

• Toast Notifications:

  • Use the react-toastify library for displaying toast notifications.
  • Ensure the toast notifications follow the application's existing design for error messages.
  • Display success and error messages appropriately based on the upload outcome.

Acceptance Criteria:

• [ ] Upload button with an upload icon is added to the ChatInput component.
• [ ] File picker dialog opens on button click.
• [ ] Image is sent to the /api/image/upload API endpoint and URL is returned.
• [ ] URL is saved and passed along with future chat messages.
• [ ] Error message is displayed using a toast notification if the upload fails.
• [ ] Image validation is implemented (under 20MB and either PNG or JPEG).
• [ ] Upload button is placed to the left of the "send message" button.
• [ ] Multiple image uploads are handled in parallel.
• [ ] Error message is shown if a non-image file is uploaded.
• [ ] Loading state is managed correctly during the upload process.
• [ ] Images are included in all future messages sent to the chat api
• [ ] Uploaded images are displayed in the chat interface immediately after upload.
• [ ] Toast notifications are used for displaying success and error messages.

Here is some research to help with your task:

Research

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Chat Interface Component Location and Structure
• Description: To implement the image upload feature, we need to identify the chat interface component, understand its structure, and determine how to integrate the new upload button and associated functionality.

2. File Structure and File Paths:

   • The chat interface component is located in the following file:
     • src/app/dashboard/[org]/[repo]/[developer]/components/chat/Chat.tsx
   • Directory Structure Overview:
     • The src/app/dashboard/[org]/[repo]/[developer]/components/chat directory contains components related to the chat interface.

3. Code Snippets:

   • Chat Interface Component (src/app/dashboard/[org]/[repo]/[developer]/components/chat/Chat.tsx):

     export const Chat: FC<Props> = ({
     messages,
     loading,
     onSend,
     onCreateNewTask,
     onUpdateIssue,
     isResponding = false,
     messagesEndRef,
     sidebarRef,
     checkIfAtBottom,
     scrollToBottom,
     isAtBottom,
     }) => (
     <div
      className="space-between flex flex-col rounded-lg px-2 pb-8 sm:p-4"
      style={{ height: "calc(100vh - 6rem)" }}
     >
      <div
        className="hide-scrollbar flex flex-1 flex-col overflow-y-auto"
        ref={sidebarRef}
        onScroll={checkIfAtBottom}
      >
        {messages.map((message, index) => (
          <div key={index} className="my-1 sm:my-2">
            <ChatMessage
              messageHistory={messages}
              message={message}
              onCreateNewTask={onCreateNewTask}
              onUpdateIssue={onUpdateIssue}
              loading={loading}
            />
          </div>
        ))}
     
        {loading && (
          <div className="my-1 sm:my-1.5">
            <ChatLoader />
          </div>
        )}
        <div ref={messagesEndRef} />
      </div>
     
      <div className="relative left-0 mt-3 w-full sm:mt-6">
        <ChatInput
          onSend={onSend}
          isResponding={isResponding}
          loading={loading}
        />
        {!isAtBottom && (
          <div
            className="absolute left-1/2 top-0 -my-14 flex h-10 w-10 -translate-x-1/2 transform cursor-pointer items-center justify-center rounded-full border border-gray-300 bg-white bg-opacity-80  transition duration-300 ease-in-out hover:bg-opacity-100"
            onClick={scrollToBottom}
          >
            <FontAwesomeIcon icon={faArrowDown} size="2x" />
          </div>
        )}
      </div>
     </div>
     );

   • The Chat component renders the chat messages and the input field.
   • The ChatInput component is responsible for handling user input and sending messages.
   • The ChatMessage component renders individual chat messages.

4. Styles and Themes:

   • The chat interface uses Tailwind CSS for styling.
   • The specific styles for the chat interface can be found in the src/styles/globals.css file and the tailwind.config.ts file.
   • The design follows a dark theme with blue and gray colors.
   • The chat interface is responsive and adapts to different screen sizes using Tailwind CSS's responsive utilities.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Existing File Handling or Upload Mechanism
• Description: This information is needed to understand if the application already has components or functions for handling file uploads. This will help determine if we can reuse existing code or need to implement a new mechanism for image uploads.

2. File Structure and File Paths:

   • Existing Files:
     • src/app/api/image/upload/route.ts: This file defines an API route for handling image uploads.
   • Relevant Directory Structure:
     • src/app/api/image: This directory contains API routes related to image handling.

3. Code Snippets:

   // src/app/api/image/upload/route.ts
   import { type NextRequest, NextResponse } from "next/server";
   import {
    getSignedUrl,
    resizeImageForGptVision,
    uploadToS3,
    IMAGE_TYPE,
   } from "~/server/utils/images";
   
   const bucketName = process.env.BUCKET_NAME ?? "";
   
   interface Body {
    image: unknown;
    imageType?: string;
    imageName?: string;
    shouldResize?: boolean;
   }
   
   export async function POST(req: NextRequest) {
    try {
      const {
        image,
        imageType,
        imageName,
        shouldResize = false,
      } = (await req.json()) as Body;
   
      if (!image || typeof image !== "string") {
        return NextResponse.json(
          {
            success: false,
            message: "Invalid image - expected base64 encoded string",
          },
          { status: 400 },
        );
      }
   
      const verifiedImageType = imageType as IMAGE_TYPE;
      if (!imageType || !Object.values(IMAGE_TYPE).includes(verifiedImageType)) {
        return NextResponse.json(
          {
            success: false,
            message: "Invalid imageType - expected image/jpeg or image/png",
          },
          { status: 400 },
        );
      }
   
      let imageBuffer = Buffer.from(image, "base64");
      if (shouldResize) {
        imageBuffer = await resizeImageForGptVision(
          imageBuffer,
          verifiedImageType,
        );
      }
      const imagePath = await uploadToS3(
        imageBuffer,
        verifiedImageType,
        bucketName,
        imageName,
      );
      const url = await getSignedUrl(imagePath, bucketName);
      return NextResponse.json({ success: true, url });
    } catch (error) {
      console.log("Error uploading image", error);
      return NextResponse.json(
        { success: false, errors: [String(error)] },
        { status: 500 },
      );
    }
   }

   • The codebase contains an API route (src/app/api/image/upload/route.ts) that handles image uploads. It receives a base64 encoded image, validates the image type and size, resizes the image if needed, uploads it to an S3 bucket, and returns a signed URL for the uploaded image.

   • Based on this information, the application already has a mechanism for handling file uploads, specifically for images. This existing code can be reused for the new image upload feature in the chat interface. However, additional information may be needed regarding how the frontend components should interact with this API route and how the returned URL should be integrated into the chat messages.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: API Interaction for File Uploads
• Description: To understand how the application currently handles API interactions, particularly for file uploads, we need to analyze the codebase for existing patterns and implementations. This information will guide the implementation of the new image upload feature for the chat interface, ensuring consistency and adherence to established conventions.

2. File Structure and File Paths:

   • File to be modified: src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatInput.tsx
   • Relevant existing file: src/app/api/image/upload/route.ts
   • Directory Structure Overview:
     • The src/app directory contains the application's frontend code.
     • The src/app/api directory houses the backend API routes.
     • The src/app/api/image/upload directory specifically handles image upload functionality.

3. Code Snippets:

   • Example of API interaction from src/app/_components/Events.tsx:

     "use client";
     
     import { api } from "~/trpc/react";
     import { TaskType } from "~/server/db/enums";
     
     export function Events() {
     const { mutate: addMutate } = api.events.add.useMutation();
     
     // subscribe to new posts and add
     api.events.onAdd.useSubscription(
      { org: "PioneerSquareLabs", repo: "t3-starter-template" },
      {
        onData(event) {
          console.log("Subscription data:", event);
        },
        onError(err) {
          console.error("Subscription error:", err);
        },
      },
     );
     
     const onAddEvent = async () => {
      addMutate({
        type: TaskType.command,
        projectId: 39,
        repoFullName: "PioneerSquareLabs/t3-starter-template",
        issueId: null,
        pullRequestId: null,
        userId: "cpirich",
        payload: {
          type: TaskType.command,
          exitCode: 0,
          command: "git clone",
          response: "done",
          directory: "/",
        },
      });
     };
     
     return (
      <div>
        <button
          onClick={onAddEvent}
          className="rounded-full bg-white/40 px-10 py-3 font-semibold no-underline transition hover:bg-white/70"
        >
          Add Event
        </button>
      </div>
     );
     }
     

   • API route for image upload from src/app/api/image/upload/route.ts:

     import { type NextRequest, NextResponse } from "next/server";
     import {
     getSignedUrl,
     resizeImageForGptVision,
     uploadToS3,
     IMAGE_TYPE,
     } from "~/server/utils/images";
     
     const bucketName = process.env.BUCKET_NAME ?? "";
     
     interface Body {
     image: unknown;
     imageType?: string;
     imageName?: string;
     shouldResize?: boolean;
     }
     
     export async function POST(req: NextRequest) {
     try {
      const {
        image,
        imageType,
        imageName,
        shouldResize = false,
      } = (await req.json()) as Body;
     
      if (!image || typeof image !== "string") {
        return NextResponse.json(
          {
            success: false,
            message: "Invalid image - expected base64 encoded string",
          },
          { status: 400 },
        );
      }
     
      const verifiedImageType = imageType as IMAGE_TYPE;
      if (!imageType || !Object.values(IMAGE_TYPE).includes(verifiedImageType)) {
        return NextResponse.json(
          {
            success: false,
            message: "Invalid imageType - expected image/jpeg or image/png",
          },
          { status: 400 },
        );
      }
     
      let imageBuffer = Buffer.from(image, "base64");
      if (shouldResize) {
        imageBuffer = await resizeImageForGptVision(
          imageBuffer,
          verifiedImageType,
        );
      }
      const imagePath = await uploadToS3(
        imageBuffer,
        verifiedImageType,
        bucketName,
        imageName,
      );
      const url = await getSignedUrl(imagePath, bucketName);
      return NextResponse.json({ success: true, url });
     } catch (error) {
      console.log("Error uploading image", error);
      return NextResponse.json(
        { success: false, errors: [String(error)] },
        { status: 500 },
      );
     }
     }
     

   • The codebase demonstrates the use of tRPC for API interactions.

   • The src/app/api/image/upload/route.ts file provides a POST endpoint for handling image uploads.

   • The uploadToS3 function suggests that images are uploaded to an S3 bucket.

   • The getSignedUrl function likely generates a signed URL for accessing the uploaded image.

4. API Contracts:

   • The /api/image/upload endpoint accepts a POST request with the following parameters:

     • image: The image data encoded as a base64 string.
     • imageType: The MIME type of the image (e.g., "image/jpeg" or "image/png").
     • imageName: (Optional) The desired name for the uploaded image.
     • shouldResize: (Optional) A boolean indicating whether the image should be resized before uploading.

   • The API response includes:

     • success: A boolean indicating whether the upload was successful.
     • url: The signed URL of the uploaded image if the upload was successful.

   • Error Handling:

     • The API returns a 400 status code for invalid image data or image type.
     • The API returns a 500 status code for internal server errors during the upload process.

5. Component Breakdown:

   • The image upload feature will likely be implemented as a new component within the chat interface.
   • This component will handle:
     • Displaying the upload button.
     • Opening the file picker dialog.
     • Validating the selected image.
     • Sending the image data to the /api/image/upload endpoint.
     • Handling the API response and updating the chat message context with the returned URL.
     • Displaying error messages using toast notifications.

6. Styles and Themes:

   • The codebase uses Tailwind CSS for styling.
   • The upload button should be styled consistently with other buttons in the chat interface.
   • The toast notifications should follow the application's existing design for error messages.
   • The component should be responsive and adapt to different screen sizes.

Based on the analysis of the codebase, the application utilizes tRPC for API interactions and provides a dedicated endpoint for image uploads. The implementation of the new image upload feature can leverage these existing patterns and conventions for consistency and maintainability. However, additional information may be needed regarding the specific implementation details of the uploadToS3 and getSignedUrl functions, as well as the desired design and styling for the new component.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Toast Notification Strategy
• Description: To maintain consistency and avoid introducing conflicting approaches, it's essential to understand how the existing codebase handles toast notifications, particularly for errors or messages. This information will guide the implementation of toast notifications for the image upload feature, ensuring a cohesive user experience.

2. File Structure and File Paths:

   • The codebase utilizes the react-toastify library for displaying toast notifications.
   • The primary file demonstrating toast notification usage is: src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatMessage.tsx.
   • Additionally, the src/app/layout.tsx file includes the ToastContainer component from react-toastify, which is necessary for rendering the notifications.

3. Code Snippets:

   // src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatMessage.tsx
   import { toast } from "react-toastify";
   
   // ...
   
   const copyToClipboard = async (text: string) => {
    await navigator.clipboard.writeText(text);
    toast.success("Copied to clipboard");
   };

   // src/app/layout.tsx
   import { ToastContainer } from "react-toastify";
   import "react-toastify/dist/ReactToastify.css";
   
   // ...
   
   export default function RootLayout({
    children,
   }: {
    children: React.ReactNode;
   }) {
    return (
      <html lang="en">
        <body
          className={`h-screen w-screen bg-[#1d265d] text-center font-sans leading-relaxed text-white ${poppins.variable}`}
        >
          <TRPCReactProvider>{children}</TRPCReactProvider>
          <ToastContainer />
        </body>
      </html>
    );
   }

   • The codebase demonstrates the use of toast.success() for displaying success messages.
   • It is likely that react-toastify also provides methods for displaying error messages, such as toast.error().
   • Further investigation into the react-toastify documentation may be needed to explore additional customization options and notification types.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: State Management in Chat Interface
• Description: Understanding how the chat interface manages state, particularly for loading states and data persistence, is crucial for implementing the image upload feature. This information will guide the integration of loading state management during image uploads and ensure the uploaded image URLs are correctly persisted and sent with subsequent chat messages.

2. File Structure and File Paths:

   • Files to be Modified:
     • src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatInput.tsx
     • src/app/dashboard/[org]/[repo]/[developer]/components/chat/index.tsx
   • Overview:
     • The chat interface is located within the src/app/dashboard/[org]/[repo]/[developer]/components/chat directory.
     • ChatInput.tsx handles user input and the "send message" button.
     • index.tsx manages the overall chat component, including message history and loading state.

3. Code Snippets:

   • Loading State Management:

     // src/app/dashboard/[org]/[repo]/[developer]/components/chat/index.tsx
     const [loading, setLoading] = useState<boolean>(false);
     
     // ...
     
     const handleSend = async (message: Message) => {
     // ...
     setLoading(true);
     // ...
     setLoading(false);
     // ...
     };

   • Data Persistence:

     // src/app/dashboard/[org]/[repo]/[developer]/components/chat/index.tsx
     const [messages, setMessages] = useState<Message[]>([]);
     
     // ...
     
     const handleSend = async (message: Message) => {
     const updatedMessages = [...messages, message];
     setMessages(updatedMessages);
     // ...
     };

   • Additional Information:

     • The codebase does not explicitly show how uploaded image URLs would be persisted or passed along with chat messages. This aspect would require further implementation and potentially involve using React Context or a state management library like Redux or Zustand.
     • The current loading state management is basic and may need to be expanded to accommodate parallel image uploads.
     • The provided code snippets demonstrate the basic principles of state management in the chat interface, but a deeper understanding of the overall application architecture and data flow might be necessary for a complete implementation of the image upload feature.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Existing File Validation Logic for Image Uploads
• Description: This information is needed to determine if there is existing validation logic for file types and sizes that can be extended for the new image upload feature. This will help to avoid redundant code and maintain consistency in the codebase.

2. File Structure and File Paths:

   • File to be modified: src/app/api/image/upload/route.ts
   • Directory structure: The src/app/api/image/upload directory contains the API route for image uploads.

3. Code Snippets:

   if (!image || typeof image !== "string") {
     return NextResponse.json(
       {
         success: false,
         message: "Invalid image - expected base64 encoded string",
       },
       { status: 400 },
     );
   }
   
   const verifiedImageType = imageType as IMAGE_TYPE;
   if (!imageType || !Object.values(IMAGE_TYPE).includes(verifiedImageType)) {
     return NextResponse.json(
       {
         success: false,
         message: "Invalid imageType - expected image/jpeg or image/png",
       },
       { status: 400 },
     );
   }

   • Coding conventions: The codebase follows standard TypeScript conventions.
   • Example of similar component: The existing code in src/app/api/image/upload/route.ts provides an example of how to validate and upload images.

The codebase contains validation logic for image types (JPEG and PNG) in the src/app/api/image/upload/route.ts file. However, there is no explicit validation for file size. Additional logic will need to be implemented to validate the image size before uploading.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Asynchronous Operations in Parallel
• Description: The user is asking how the application handles multiple asynchronous operations in parallel. This information is needed to understand the existing mechanisms for managing concurrent tasks, particularly in the context of uploading multiple images simultaneously.

2. File Structure and File Paths:

   • The codebase does not provide explicit examples of handling multiple asynchronous operations in parallel. Further investigation is required to determine the specific implementation.

3. Code Snippets:

   • The codebase lacks specific code snippets directly demonstrating parallel asynchronous operations. More context or specific files related to image upload or asynchronous task management are needed to provide relevant examples.

   I lack the necessary information to fully address the user's query. The codebase does not contain explicit examples or explanations of how parallel asynchronous operations are handled. To provide a more comprehensive answer, additional information is needed, such as:

   • Specific files or components related to image upload functionality.
   • Code sections dealing with asynchronous task management or queuing.
   • Documentation or comments explaining the application's approach to concurrency.

Response from ResearchCodebase: 1. Detailed Specifications:

• Title: Visual Representation of Loading States
• Description: This information is needed to understand how loading states are visually represented in the application, which is necessary to maintain consistency when implementing a new loading state for the image upload feature.

2. File Structure and File Paths:

   • Files to be modified:
     • src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatInput.tsx
     • src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatMessage.tsx
     • src/app/dashboard/[org]/[repo]/[developer]/components/todos/index.tsx
   • Overview of directory structure: The relevant files are located within the components directory, which is further organized into subdirectories for different parts of the application, such as chat and todos.

3. Code Snippets:

   • Example 1 (ChatInput.tsx):

     <div
     className={`flex w-full max-w-4xl flex-col items-start rounded-lg border border-gray-600 p-4 backdrop-blur-md ${
      isResponding || loading ? "opacity-50" : ""
     }`}
     >

     This code snippet shows that the ChatInput component applies an opacity-50 class to the input area when the loading state is true. This suggests that a semi-transparent overlay might be used to indicate a loading state.

   • Example 2 (ChatMessage.tsx):

     <div
     className={`inline-flex  items-center justify-center gap-2 rounded border border-gray-400 bg-white px-6 py-2 ${loading ? "cursor-wait opacity-50 " : "cursor-pointer "}`}
     onClick={() => onCreateNewTask(messageHistory)}
     >
     <div className="text-center text-xs font-medium text-black">
      {loading ? "Creating Issue..." : "Create New Issue"}
     </div>
     </div>

     This code snippet from the ChatMessage component demonstrates that a button is visually modified during a loading state. The button's text changes to "Creating Issue..." and the opacity-50 class is applied. Additionally, the cursor-wait class is used to indicate that the button is in a loading state.

   • Example 3 (todos/index.tsx):

     <div className="flex h-screen flex-col items-center justify-center space-x-2 space-y-4 border-x border-coolGray-400/20 bg-gray-900 bg-slate-50/5 text-2xl text-blueGray-500">
     <div>Loading...</div>
     <div className="flex flex-row items-center justify-center space-x-2">
      <div className="h-6 w-6 animate-bounce rounded-full bg-light-blue [animation-delay:-0.3s]"></div>
      <div className="h-6 w-6 animate-bounce rounded-full bg-pink [animation-delay:-0.15s]"></div>
      <div className="h-6 w-6 animate-bounce rounded-full bg-orange"></div>
     </div>
     </div>

     This code snippet from the todos/index.tsx file showcases a different visual representation of a loading state. It displays the text "Loading..." along with three animated dots that bounce using the animate-bounce class.

   • Coding Conventions: The codebase follows standard React and Tailwind CSS conventions.

   • Similar Components: The ChatLoader component in src/app/dashboard/[org]/[repo]/[developer]/components/chat/ChatLoader.tsx provides another example of a loading state, using animated dots with different colors.

   • Additional Information: The codebase provides several examples of how loading states are visually represented. However, it is unclear if there are any specific design guidelines or themes that should be followed for loading states. Further clarification or design specifications might be needed to ensure consistency in the implementation of the new image upload feature's loading state.

@jacob-ai-bot

[jacob-local-kevin[bot]]

JACoB here...

You mentioned me on this issue and I am busy taking a look at it.

I'll continue to comment on this issue with status as I make progress.

[jacob-local-kevin[bot]]

Update

I've completed my initial work on this issue and have created a pull request: JACoB PR for Issue Add Image Upload to Chat to allow for creating code from screenshots.

The changes currently result in an error, so I'll be making some additional changes before it is ready to merge.