MacPaw / OpenAI

Swift community driven package for OpenAI public API
MIT License
2.09k stars 351 forks source link

Explain Image functionality #148

Open leventmolla opened 9 months ago

leventmolla commented 9 months ago

Discussed in https://github.com/MacPaw/OpenAI/discussions/147

Originally posted by **leventmolla** January 25, 2024 OpenAI has a new model gp4-1106-vision-preview which can explain a collection of images. I think it could be done with the general chat completions endpoint, but the documentation is not very clear about the message structure. There should be a text prompt initially describing the task and follow-up messages that contain the images. I tried this and passed base64-encoded images, got errors (for some reason the number of tokens requested is a very large number and the query fails). I then tried to pass the URLs of the files for the images, which failed as well. So I am at a loss about how to use this functionality.
kalafus commented 9 months ago

Correct, this is done with the ChatCompletions endpoint. The documentation indicates that the User role can send 3 types of Message Content:

https://github.com/MacPaw/OpenAI/pull/169 adds the ChatCompletionContentPartImageParam type (as it's called in the Python API). I hadn't included base64 support, but can do so now.

kalafus commented 9 months ago

looking at the Python code, OpenAI counts on you to encode image data to base64 string and feed that to the image_url.url parameter; looks like i had coded it that way because I was following the Python API precisely.

class ImageURL(TypedDict, total=False): 
    url: Required[str]
    """Either a URL of the image or the base64 encoded image data.""" 

    detail: Literal["auto", "low", "high"]
    """Specifies the detail level of the image. 

    Learn more in the
    [Vision guide](https://platform.openai.com/docs/guides/vision/low-or-high-fidelity-image-understanding).
    """

class ChatCompletionContentPartImageParam(TypedDict, total=False):
    image_url: Required[ImageURL]

    type: Required[Literal["image_url"]]
    """The type of the content part."""

see src/openai/types/chat/chat_completion_content_part_image_param.py