Add support for GPT-4 with Vision

spring-projects / spring-ai

An Application Framework for AI Engineering

https://docs.spring.io/spring-ai/reference/1.0-SNAPSHOT/index.html

Apache License 2.0

2.47k stars 585 forks source link

Add support for GPT-4 with Vision #144

Open ghillert opened 6 months ago

ghillert commented 6 months ago

Having experimented with OpenAI's GPT-4 with Vision API, it would be amazing if Spring AI adds support for image-based input data (e.g. photos). This API allows you to post:

One or more images
Images are represented as base64 encoded data or as image URL

The API is located at: API reference: https://platform.openai.com/docs/guides/vision

An implementation could possibly also provide additional pre-processing in order to add support for video data as described here:

https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding

markpollack commented 6 months ago

Pull requests welcome.

ghillert commented 6 months ago

It looks like this feature would depend on https://github.com/TheoKanning/openai-java/issues/397

markpollack commented 5 months ago

We have moved away from depending on that library and will provide our own implementations. ATM, brainstorming on having the API incorporate more than just chat models. Will be able to share a spike branch shortly. Was just circling back to issues filed wrt to other AI models. And happy new year Gunnar!

arnokoehler commented 2 months ago

How is the spike branch going? Is there a way others could help contribute?

Mr-LiuDC commented 2 months ago

Same issue. Is there any progress?