Add support for Vision API

[lectrician1]

Adds the ability to use the vision API in the client. Resolves #488 and #473

Message interface:

Editing interface:

• [X] Add image to blob (base64) conversion when a file is uploaded, and store that base64 in the browser (localstorage). Not perfect and localstorage might run out of space but I don't have any other better solutions to keep the images you store from previous chats and IndexedDB looks too hard to implement.
• [X] Add image detail selection for users to select which detail their image should be processed at. see: https://platform.openai.com/docs/guides/vision/low-or-high-fidelity-image-understanding
• [X] Add a new data structure to differentiate models based on the types of inputs they can take and update editor to check if model supports images or not. I added a const modelTypes and check for it in EditView.tsx
• [X] Clean up the image upload UI (it's not perfect, but works)
• [ ] Need to make some data structure version migration function so that clients with old data structures will get theirs updated (I need help with this)

Notable changes

This PR changes the the data structures of ChatInterface, MessageInterface, and creates ContentInterface, TextContentInterface and ImageContentInterface so that it follows the updated API JSON structure for prompts that contain images.

Before:

export interface MessageInterface {
  role: Role;
  content: string;
}

After:

export type Content = 'text' | 'image_url';

export interface ImageContentInterface extends ContentInterface {
  type: 'image_url';
  image_url: {
    url: string;
  }
}

export interface TextContentInterface extends ContentInterface {
  type: 'text';
  text: string;
}

export interface ContentInterface {
  type: Content;
}

export interface MessageInterface {
  role: Role;
  content: ContentInterface[];
}

The url parameter stores the URL of the image locally (the blob: URL) and at generation-time the client converts all the blob URLs into base64 for the API to take in.

[OldKrab]

Now yarn run build fail with type errors

[Ahmet-Dedeler]

Has there been any updates with this?

[lectrician1]

Uhhh well haven't had time to finish this up but maybe will try this weekend...

[Ahmet-Dedeler]

@lectrician1 That would be amazing, this feature would literally be the revolutionary feature for this repo.

Lately the repo hasn't been very active, so this might just change the future of how active and innovative it becomes.

Please keep us updated here, will try to help :)

[lectrician1]

@Ahmet-Dedeler it's almost ready.

I just need help from @ztjhz @akira0245 or @ayaka14732 to write the migration code in migrate.ts and chat.ts to migrate the old data structure to the new one for ChatInterface and associated interfaces (see the PR description). I tried doing this and it's really confusing. I'm not sure how the whole migrate process works.

[ztjhz]

I'll take a look at this.

[Ahmet-Dedeler]

Sorry for tagging but did any of you chance to look at this @ztjhz @akira0245 @ayaka14732 ?

[Ahmet-Dedeler]

Sorry for tagging but did any of you chance to look at this @ztjhz @akira0245 @ayaka14732 ?

[AI-Maria]

@lectrician1

Is content wrong with no vision model? [{'role': 'system', 'content': [{'type': 'text', 'text': 'You are assistant'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': '1'}]}]

i asked ChatGPT and it write migration, it is works for me:

export const migrateV8 = (persistedState: LocalStorageInterfaceV7oV8) => {
  persistedState.chats.forEach((chat) => {
    chat.messages.forEach((message) => {
      // Check if the old content structure exists
      if (typeof message.content === 'string') {
        // Convert the old content string to the new content array structure
        message.content = [{
          type: 'text', // assuming all old content is of type 'text'
          text: message.content
        }] as ContentInterface[];
      }
    });
  });
};

I have error in EditView.tsx:246 state.chats is undefined, i just make this: const model = useStore((state) => state.chats != undefined ? state.chats![state.currentChatIndex].config.model : ""); maybe you found better solution