search

spring-projects / spring-ai

An Application Framework for AI Engineering
https://docs.spring.io/spring-ai/reference/index.html
Apache License 2.0
3.33k stars 856 forks source link

Add support for GPT-4 with Vision #144

Closed ghillert closed 4 months ago

ghillert commented 11 months ago

Having experimented with OpenAI's GPT-4 with Vision API, it would be amazing if Spring AI adds support for image-based input data (e.g. photos). This API allows you to post:

The API is located at: API reference: https://platform.openai.com/docs/guides/vision

An implementation could possibly also provide additional pre-processing in order to add support for video data as described here:

https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding

markpollack commented 11 months ago

Pull requests welcome.

ghillert commented 11 months ago

It looks like this feature would depend on https://github.com/TheoKanning/openai-java/issues/397

markpollack commented 10 months ago

We have moved away from depending on that library and will provide our own implementations. ATM, brainstorming on having the API incorporate more than just chat models. Will be able to share a spike branch shortly. Was just circling back to issues filed wrt to other AI models. And happy new year Gunnar!

arnokoehler commented 7 months ago

How is the spike branch going? Is there a way others could help contribute?

Mr-LiuDC commented 7 months ago

Same issue. Is there any progress?

ThomasVitale commented 4 months ago

It should be possible to close this issue since multimodality support has been introduced in Spring AI 1.0.0-M1, including the possibility to pass images as input to OpenAI: https://docs.spring.io/spring-ai/reference/api/multimodality.html

Example: https://github.com/ThomasVitale/llm-apps-java-spring-ai/tree/main/02-prompts/prompts-multimodality-openai

markpollack commented 4 months ago

Closing the issue. Thanks Thomas, issue pruning has been overdue, working through it now.