[Feature]: Support https://... pdf files for vertex ai

[asleepynerd]

What happened?

sending a request with this:

{
    "model": "g1.5",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "what does this pdf say?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.clickdimensions.com/links/TestPDFfile.pdf"
                }
            }
        ]
      }
    ]
  }

Gemini 1.5 / gemini 1.5 flash

Using the /v1/chat/completions endpoint

Relevant log output

{
    "error": {
        "message": "VertexAIException - cannot identify image file <_io.BytesIO object at 0x701ec5321850>\nmodel: gemini-1.5-pro-preview-0409\nmodel_group: g1.5\ndeployment: gemini-1.5-pro-preview-0409\nvertex_project: heyobro\nvertex_location: us-central1\n",
        "type": null,
        "param": null,
        "code": 500
    }
}

Twitter / LinkedIn details

twitter: @ellsiecodes - https://x.com/ellsiecodes

[krrishdholakia]

@joshiewtf i believe vertex pdf needs to be on gs://

this is the comparable vertex example:

await chat.sendMessageStream([
  {
    text: 'Where is this candidate based?'
  },
  {
    fileData: {
      mimeType: 'application/pdf',
      fileUri: 'gs://resume.pdf'
    }
  }
]);

do you have http pdf files working with vertex ai today? would help to see a working code example @joshiewtf

[asleepynerd]

Docs don't say anything here about it:

https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.tuningJobs#Content

@krrishdholakia

[krrishdholakia]

looking here - https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#non-stream-multi-modality

it says the file uri is on google cloud storage

if you can share a working call with pdf, that would help @letmefocus