search

vercel / ai

Build AI-powered applications with React, Svelte, Vue, and Solid
https://sdk.vercel.ai/docs
Other
9.38k stars 1.37k forks source link

Pass `detail` parameter to `ImagePart` in `CoreUserMessage` #2355

Closed yannxaver closed 1 week ago

yannxaver commented 1 month ago

Feature Description

OpenAI exposes a detail parameter for images. (https://platform.openai.com/docs/guides/vision/low-or-high-fidelity-image-understanding)

Is it possible to set this parameter using the ai library?

Use Case

No response

Additional context

No response

lgrammel commented 1 month ago

Not currently. This is very openai specific.

Do you have a use case that is not possible with the default automatic detail setting?

yannxaver commented 1 month ago

Sometimes it's preferable to set detail to low to reduce input tokens (costs). If it is set to automatic OpenAI will choose the detail level based on the image size. So if I can't set the detail level manually, I would have to resize the image instead to activate detail=low. That would take additional resources and be slower.

angelhodar commented 1 month ago

I am investigating to include the ai sdk in my current workflow but this is a blocker because we dont want to manually downsize all images needed in our prompts (usually 20+). Also in gpt-4o-mini the token consumption is much more than with base gpt-4o because they want to make the same pricing in both, so being able to set detail to low significantly reduces it

lgrammel commented 1 month ago

When you use detail: "low", do you set it for all images, or do you change this setting only for certain images?

angelhodar commented 1 month ago

I personally use it for all images

lgrammel commented 1 week ago

WIP PR: https://github.com/vercel/ai/pull/2918

lgrammel commented 1 week ago

https://sdk.vercel.ai/providers/ai-sdk-providers/openai#image-detail