Assistants V2 File Annotations Replacement

[jackitaliano]

Scenario

• deep-chat-dev v9.0.174
• directConnection to OpenAI Assistants V2

It seems the sandbox:... URLs are not being replaced by the file contents found at the given file_id in the annotations (e.g. data:image...).

I have verified that the messages in question do have annotations present via outside requests for their contents.

Recreate

there are other ways to recreate, this one seems most consistent

• Ask assistant to give example data for a csv and create a suitable bar chart for it (in one message)
• One text response is given with the corresponding bar chart image, and a link to download.
• Both are their respective sandbox:... URLs rather than data:image... URLs

can also make this happen by uploading your own csv and then asking for the same

Thoughts

We have our own logic that loads/restores chat history on page load. The only changes we've had to make to account for V2 updating to attachments instead of file_ids in the response on file retrieval and any parsing of messages with files. Along the lines of:

// Let message be a message object returned from OpenAI

// old version
const fileIds = message.file_ids 

// new version
const fileIds = message.attachments.map(attachment => attachment.file_id);

I'm not sure if this is where the problem lies with deep-chat, but I don't believe there was anything else changed to the Assistants API relevant to this specific issue.

Glad to see you're still keeping up with what you can for deep-chat 😄 (I was absent for a few weeks)

[OvidijusParsiunas]

Hey @jackitaliano.

Thankyou for letting me know about the issue!

I was able to reproduce the issue and have managed to fix it for the default non-stream API. This fix is available in deep-chat-dev and deep-chat-react-dev versions 9.0.179. Let me know if there are any issues.

I can see that this issue is still persistent in the stream API which I will take care of the next day.

Stay tuned!

[jackitaliano]

This seems to have fixed the issue.

I also see that you put in some logic to prevent files from being displayed if they're already annotated, which is appreciated as well.

Thank you!

[OvidijusParsiunas]

Good to hear it fixed the issue.

The reason why I was also returning the files was I believe I was trying to replicate the OpenAI Assistant Playground behaviour as it was doing that as well, but I don't believe it does any more. So everything looks clean now.

[OvidijusParsiunas]

File annotations have now been fixed for streamed messages in deep-chat-dev and deep-chat-react-dev version 9.0.180.

[jackitaliano]

Good to hear it fixed the issue.

The reason why I was also returning the files was I believe I was trying to replicate the OpenAI Assistant Playground behaviour as it was doing that as well, but I don't believe it does any more. So everything looks clean now.

This makes sense. Appreciate the quick fixes and updates!