[Question] Azure OpenAI - Latest API version missing "image" message content for Assistant API

hayescode commented 2 months ago

API Spec link

https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI

API Spec version

2024-07-01-preview

Question/Query

Message Content

When trying to add non-text messages we get an error message indicating only "text" is allowed. OpenAI has supported this for a couple months so it's surprising to see this missing still. Do we have an ETA on when this will be added?

await client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content=[{'text': 'what does this say?', 'type': 'text'}, {'image_url': {'url': 'data:image/png;base64,iVBORw0KG...', 'detail': 'auto'}, 'type': 'image_url'}]
)

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid message content: only messages with 'type=text' are supported currently.", 'type': 'invalid_request_error', 'param': 'content', 'code': 'invalid_type'}}

Environment

No response

hao1300 commented 2 months ago

We recently ran into this issue as well? Any update on this?

wilson101xx commented 2 months ago

Same here any updates?

hayescode commented 3 weeks ago

@jpalvarezl @trrwilson Will this be fixed in #30909 ? I've tried 2024-08-01-preview and 2024-09-01-preview but I still get the same error.

sorieux commented 3 weeks ago

I’m also encountering this issue with 2024-09-01-preview, and it’s blocking progress on one of my projects. Are there any workarounds available to enable using images with an assistant?

hayescode commented 1 week ago

I'm guessing it comes from here, which is using the 2024-02-15 version of Assistants. Azure OpenAI is falling far behind OpenAI in many cases it seems they can't maintain parity :(

rrosztoczy commented 15 hours ago

I'm also waiting for this.

Azure / azure-rest-api-specs