Assistant Message and image_url

Lambdua / openai4j

Java client library for OpenAI API.Full support for all OpenAI API models including Completions, Chat, Edits, Embeddings, Audio, Files, Assistants-v2, Images, Moderations, Batch, and Fine-tuning.

MIT License

303 stars 25 forks source link

Assistant Message and image_url #44

Closed chrisetheridge closed 3 months ago

chrisetheridge commented 3 months ago

When using this code: https://github.com/Lambdua/openai4j/blob/61fa82c77cf28653d0c5106c8a92def7d038388b/service/src/test/java/com/theokanning/openai/service/assistants/AssistantImageTest.java#L64-L87

The messages returned from listMessages don't have an ImageUrl object. They only have a MessageContent object with type image_url but no image_url. The Assistant Playground does show that this data is returned from the API when listing thread messages.

Is it possibly because the messages returned from the Threads API are being coerced to Assistant MessageContent which doesn't cater for image_url?

chrisetheridge commented 3 months ago

For example:

Lambdua commented 3 months ago

I'll take a look later. It may be a deserialization problem.

chrisetheridge commented 3 months ago

Thank you!

I think its because the response is coerced to an Assistant Message, here: https://github.com/Lambdua/openai4j/blob/57b58f23ba3ec127c1c9d214ea86177b8ccd4d30/client/src/main/java/com/theokanning/openai/client/OpenAiApi.java#L218

But, the Assistant Message doesn't support image_url images.

Lambdua commented 3 months ago

I haven't had time to look at it these past two days. It will be next week. If you are in a hurry, you can submit your mq to the specifications. Thank you.

chrisetheridge commented 3 months ago

No problem, no hurry.

I opened a PR here of a fix: https://github.com/Lambdua/openai4j/pull/45 . Can delete if you'd like to do something different!