Downloading an Image File by File ID in Azure OpenAI Assistant using JavaScript SDK

[RohitMungi-MSFT]

Is your feature request related to a problem? Please describe. Downloading an Image File by File ID in Azure OpenAI Assistant using JavaScript SDK

Describe the solution you'd like A sample to show how an image file can be downloaded locally using Azure OpenAI assistant. Similar to the one shown in this python SDK sample. https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/Assistants/api-in-a-box/wind_farm/assistant-wind_farm.ipynb

Describe alternatives you've considered Using assistantsClient.listMessages to list the messages and filtering by item.type === "image_file" only gives image file id and there is no option to download the file or give the file URL.

const runMessages = await assistantsClient.listMessages(assistantThread.id);
for (const runMessageDatum of runMessages.data) {
  for (const item of runMessageDatum.content) {
    if (item.type === "text") {
      console.log(item.text.value);
    } else if (item.type === "image_file") {
      console.log(item.imageFile.fileId);
    }
  }
}

Additional context Using getAssistantFile also does not have an option to get the image link.

See this Q&A thread for additional details.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[deyaaeldeen]

Hi @RohitMungi-MSFT,

Thanks for opening this report! We moved the Azure OpenAI client to the official OpenAI client library for JavaScript instead. Checkout the updated samples. To answer your specific question, you can access the file URL as follows:

  for await (const runMessageDatum of runMessages) {
    for (const item of runMessageDatum.content) {
      switch (item.type) {
        case "text": {
          console.log(`${runMessageDatum.role}: ${item.text.value}`);
          break;
        }
        case "image_file": {
          console.log(`Received image: ${item.image_file.file_id}`);
          break;
        }
        case "image_url": {
          console.log(`Received image: ${item.image_url.url}`);
          break;
        }
        default: {
          console.log(`Unhandled item type: ${item.type}`);
        }
      }
    }
  }

[github-actions[bot]]

Hi @RohitMungi-MSFT. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[SuMyatHlaing30]

I can not generate image file with image_url. I am testing as below but assistant does not provide image url . please see below code and log.

`
const { AssistantsClient, AzureKeyCredential } = require("@azure/openai-assistants");
const endpoint = "..";
const azureApiKey = "..";
const assistantsClient = new AssistantsClient(endpoint, new AzureKeyCredential(azureApiKey));

async function main() {
      const assistantResponse = await assistantsClient.createAssistant({
      model: "gpt-4o",
      name: "Graph Test",
      instructions: "You are a helpful assistant",
      tools: [{ type: "code_interpreter" }],
    });
    console.log("assistantResponse",assistantResponse);
    const assistantThread = await assistantsClient.createThread({});
    //console.log("assistantThread",assistantThread);

    const question = "provide sample graph";
    const threadResponse = await assistantsClient.createMessage(
      assistantThread.id,
      "user",
      question,
    );
    //console.log("threadResponse",threadResponse);
    let runResponse = await assistantsClient.createRun(
        assistantThread.id,
        {
          assistantId: assistantResponse.id,
          instructions: "Please address the user as Jane Doe. The user has a premium account.",
        }
      );

    do {
      await new Promise((r) => setTimeout(r, 500));
      runResponse = await assistantsClient.getRun(assistantThread.id, runResponse.id);
      const runSteps = await assistantsClient.listRunSteps(
        assistantThread.id,
        runResponse.id,
        {
          requestOptions: {},
          limit: 1,
        },
      );
      //console.log(runSteps);
    } while (runResponse.status === "queued" || runResponse.status === "in_progress");

    const runMessages = await assistantsClient.listMessages(assistantThread.id);
    for (const runMessageDatum of runMessages.data) {
      for (const item of runMessageDatum.content) {
        console.log("Item ....",item);
        switch (item.type) {
          case "text": {
            console.log(`${runMessageDatum.role}: ${item.text.value}`);
            break;
          }
          case "image_file": {
            console.log(`Received image: ${item.imageFile.fileId}`);
            break;
          }
          case "image_url": {
            console.log(`Received image: ${item.image_url.url}`);
            break;
          }
          default: {
            console.log(`Unhandled item type: ${item.type}`);
          }
        }
      }
    }
  }
main().catch((err) => {
    console.error("The sample encountered an error:", err);
  });

  module.exports = { main };`

**> > assistantResponse {

id: 'asst_...', createdAt: 2024-09-10T00:42:54.000Z, name: 'Graph Test', description: null, model: 'gpt-4o', instructions: 'You are a helpful assistant', tools: [ { type: 'code_interpreter' } ], fileIds: [], metadata: {} } Item .... { type: 'image_file', text: {}, imageFile: { file_id: 'assistant-cXCVxZQAgmpyxdWwm9mrGmAi' } } Received image: undefined Item .... { type: 'text', text: { value: 'Here is a sample graph displaying the sine and cosine functions. The x-axis represents values in radians from 0 to \(2\pi\), while the y-axis represents the corresponding sine and cosine values. The grid and legend help to differentiate between the two functions.\n' + '\n' + 'If you have any specific requirements for a graph, please let me know!', annotations: [] }, imageFile: {} } assistant: Here is a sample graph displaying the sine and cosine functions. The x-axis represents values in radians from 0 to (2\pi), while the y-axis represents the corresponding sine and cosine values. The grid and legend help to differentiate between the two functions.

If you have any specific requirements for a graph, please let me know! Item .... { type: 'text', text: { value: "Certainly, Jane Doe. Here is a sample graph generated using Python's matplotlib library. This graph will show a simple plot of the sine and cosine functions.\n" + '\n' + "Let's create and display the graph:", annotations: [] }, imageFile: {} } assistant: Certainly, Jane Doe. Here is a sample graph generated using Python's matplotlib library. This graph will show a simple plot of the sine and cosine functions.

Let's create and display the graph: Item .... { type: 'text', text: { value: 'provide sample graph', annotations: [] }, imageFile: {} } user: provide sample graph**

[deyaaeldeen]

@RohitMungi-MSFT Thanks for sharing. Could I ask you to migrate to using openai instead of @azure/openai-assistants? The migration guide has examples for how to do so.

[SuMyatHlaing30]

@deyaaeldeen I have migrated and tested like that codeintepreter but still show like below. not generating image url. I want assistant to visualize graph.

for await (const runMessageDatum of runMessages) {
    for (const item of runMessageDatum.content) {
      switch (item.type) {
        case "text": {
          console.log(`${runMessageDatum.role}: ${item.text.value}`);
          break;
        }
        case "image_file": {
          console.log(`Received image image_file: ${item.image_file.file_id}`);
          break;
        }
        case "image_url": {
          console.log(`Received image image_url: ${item.image_url.url}`);
          break;
        }
        default: {
          console.log(`Unhandled item type: ${item.type}`);
        }
      }
    }
  }`

Received image image_file: assistant-plQBx6NrLhhuXJ5mtS09Kkuq assistant: Here's a sample graph plotting the function ( \sin(x) ). If you have any specific data or functions you would like to visualize, let me know, Jane Doe! user: provide sample graph

[deyaaeldeen]

@RohitMungi-MSFT ah sorry I misunderstood the ask earlier. You'll need to pass the file ID to https://platform.openai.com/docs/api-reference/files/retrieve-contents. You can do so through the JS client by calling client.files.content.

[SuMyatHlaing30]

@deyaaeldeen Thank you for your support. I successfully downloaded the file using the instructions from this link.

I would like to confirm whether I need to change all my Azure OpenAI JavaScript SDK code (e.g., @azure/openai, @azure/openai-assistants) to the OpenAI SDK (openai) as part of the migration. Is that correct?

[deyaaeldeen]

@deyaaeldeen Thank you for your support. I successfully downloaded the file using the instructions from this link.

Yay, glad it worked!

I would like to confirm whether I need to change all my Azure OpenAI JavaScript SDK code (e.g., @azure/openai, @azure/openai-assistants) to the OpenAI SDK (openai) as part of the migration. Is that correct?

Yes, that's right.

[github-actions[bot]]

Hi @RohitMungi-MSFT, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.