Support files (images, csv, pdf) returned by OpenAI Assistants API Code Interpreter

[bruffridge]

I uploaded an excel file, and asked it to generate a chart of the data. I got the message "Error, please try again" in the UI. Upon inspection, I have found the error.

TypeError: Cannot read properties of undefined (reading 'value')

on this line: https://github.com/OvidijusParsiunas/deep-chat/blob/4d0af0696aee65eedf0051a9d38b24ecebc2680e/component/src/services/openAI/openAIAssistantIO.ts#L122

Problem is due to content[0].text.value which in this case contains an image file. The text response is in content[1].

lastMessage.content ==

[
    {
        "type": "image_file",
        "image_file": {
            "file_id": "file-5Hzoocp5DNP1MsbJhtY9SHjx"
        }
    },
    {
        "type": "text",
        "text": {
            "value": "The graph depicted above illustrates the flow of fuel over time, expressed in kilograms per hour, commencing from the 'Starting DateTime'. Each point on the graph corresponds to the rate at which fuel was flowing at the onset of the measurement interval.\n\nIf you require further analysis or more specific visualizations, such as highlighting distinct intervals or comparing different conditions, please provide additional instructions.",
            "annotations": []
        }
    }
]

[OvidijusParsiunas]

This is very interesting, I didn't know the API had the functionality to reference image files. I will certainly update Deep Chat to reflect this.

I personally have been having a hard time in uploading csv/xlsx files to the assistants API as it either says that the format is invalid or it is not compatible with the myfiles_browser tool I have. Did you do anything special to generate your excel file (did you use any tools), is it also a file you have uploaded before or did you ask the question along with a file.

It is currently very late where I live, so I will be able to create a fix for you tomorrow. Thanks.

[OvidijusParsiunas]

Ok, so it turns out I need to activate the Code Interpreter on the Assistant to generate images and use a prompt with something like: "Can you please give me a simple example of a scatter plot. Please save as a png"

I have made the necessary changes and released them in the deep-chat-dev and deep-chat-react-dev packages version 9.0.108. These packages work exactly the same as the normal ones, it is just that their names are different. I will promote these changes to the normal packages later next week.

To note, when I was testing out image generation, it appears that the text responses can contain a hyperlink with text like "Download the scatter plot image":

This hyperlink does not actually work as it is an unusable:

You can perhaps instruct the model to not provide a hyperlink in the responses if this is an issue for you.

Let me know if this works for you. Thanks!

[bruffridge]

I personally have been having a hard time in uploading csv/xlsx files to the assistants API as it either says that the format is invalid or it is not compatible with the myfiles_browser tool I have. Did you do anything special to generate your excel file (did you use any tools), is it also a file you have uploaded before or did you ask the question along with a file.

I didn't do anything special, just uploaded the excel file I downloaded from here.

I have made the necessary changes and released them in the deep-chat-dev and deep-chat-react-dev packages version 9.0.108. Let me know if this works for you. Thanks!

Works great, thanks so much for the fast fix! Where does the code for deep-chat-dev live? Was curious to see your code fix, but didn't see the commit in this repo.

[OvidijusParsiunas]

The code for this change can be found in this commit.

[bruffridge]

Looks like in addition to images, Code Interpreter can output csv and pdf files. Would it be possible to add support for these as well as images? These files are sent in a different json format.

• Types of files returned by Code Interpreter and JSON formats: https://platform.openai.com/docs/assistants/tools/reading-images-and-files-generated-by-code-interpreter
• Example of how to replace faulty sandbox:/ links https://platform.openai.com/docs/assistants/how-it-works/message-annotations
• API to download a file's contents with given ID: https://platform.openai.com/docs/api-reference/files/retrieve-contents

Thread about this issue, and comment from OpenAI staff: https://community.openai.com/t/how-do-download-files-generated-in-ai-assistants/493516/11

Once you have retrieved the file contents as a string you can add a download link with something like this in javascript:

let urlEncodedCSVString = encodeURI(csvContentsAsString)

<a href="data:text/csv,{urlEncodedCSVString}">download csv file</a>

For images maybe just strip out any download links.

Do you think this should go in deep-chat? I may be able to handle this on my end in responseInterceptor?

[OvidijusParsiunas]

Hi @bruffridge. Just wanted to let you know that I saw this issue and will be working on it. Unfortunately today I was travelling so did not get a chance, but I will look into it tomorrow. Thanks!

[OvidijusParsiunas]

I have updated the code to now allow the use of csv and pdf files. This code is currently available in the deep-chat-dev and deep-chat-react-dev packages version 9.0.109. It will be promoted to the main packages later this week. Thankyou!

[OvidijusParsiunas]

Hi, this has now been released in deep-chat and deep-chat-react package versions of 1.4.7.

[bruffridge]

CSV and PDF works great, thanks! However, images no longer works. It's weird but OpenAI decided that image_file should be a separate object in content and a sibling to text type object. However, pdfs and csvs are inside the text type object's text.annotations.

Image files

 "content": [
  {
    "type": "image_file",
    "image_file": {
      "file_id": "file-Yps9aQnH1RG6aKrs7dXLCb3w"
    }
  },
  {
    "type": "text",
    "text": {
      "value": "Here is an example of a scatterplot. This plot displays the relationship between two sets of randomly generated data points on the X-axis and Y-axis, respectively. Each point on the plot represents a single data point from the datasets.",
      "annotations": []
    }
  }
],

CSV and PDFs

"content": [
  {
    "type": "text",
    "text": {
      "value": "I have created a small example CSV file with some sample data. You can download it using the link below:\n\n[Download the example CSV file](sandbox:/mnt/data/example.csv)",
      "annotations": [
        {
          "type": "file_path",
          "text": "sandbox:/mnt/data/example.csv",
          "start_index": 138,
          "end_index": 167,
          "file_path": {
            "file_id": "file-Wmps4BJ2USXXdhDx3x1RtWvy"
          }
        }
      ]
    }
  }
]

[bruffridge]

When you click on the link to download the file, the filename it defaults to is download with no file extension, at least on Chrome on my Mac. Would it be possible to have the filename default to the filename inside annotations.text? It appears that deep-chat is already extracting the filename from annotations.text and displaying it as a message (example.csv and sample_report.pdf in the image below)

[OvidijusParsiunas]

Hi, the images fix has been deployed to deep-chat-dev and deep-chat-react-dev packages version 9.0.114. It will be included in the next core package release.

In regards to the file that is downloaded from the hyperlink, unfortunately the name cannot be changed as this the hyperlink is constructed via the remarkable library.