[Question]: How can I send an attachment to openai like langchain

[mballav]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I have a need to find out if a PDF is digitally or manually signed or not.

I tried sending images of pdf pages to Gpt-4o using llama-index. It could tell if the document was manually signed or not, but failed to tell me if it was digitally signed as it needed to examine the metadata of the PDF itself.

In order to do that, I would like to upload to OpenAI the PDF and then ask the same question. Langchain provides a way to upload a file as an attachment to a user message. Is that something I can do using llama-index as well?

[logan-markewich]

I think that would be using openai assistants right? Our integration for that is a tad out of date (openai has changed the api a ton), but might still work for this case

from llama_index.agent.openai import OpenAIAssistantAgent

agent = OpenAIAssistantAgent.from_new(
    name="SEC Analyst",
    instructions="You are a QA assistant designed to analyze sec filings.",
    openai_tools=[{"type": "retrieval"}],
    instructions_prefix="Please address the user as Jerry.",
    files=["data/10k/lyft_2021.pdf"],
    verbose=True,
)

[mballav]

@logan-markewich, How will your answer change if I am using an Azure deployment of Gpt-4 or Gpt-4o?

[logan-markewich]

@mballav does azure support assistants? Thats a little unexplored on our end tbh

[logan-markewich]

looking at the source code, its maybe possible if you pass in your own openai client

[logan-markewich]

You'd have to skip from_new and use the class constructor directly (and probably copy some of what from_new is doing)

[mballav]

@logan-markewich, Thanks for the replies. Could you please provide code to instantiate a client? Do I do that using the AzureOpenAI class?

[logan-markewich]

https://github.com/run-llama/llama_index/blob/0d285c690e884afd6269bc278d3f29dca75630a3/llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/openai_assistant_agent.py#L209

Probably from_new and from_existing should let you pass in the client rather than hardcoding

[logan-markewich]

@mballav yea, from openai import AzureOpenAI for the client