How to pass PDFs to Gemini using the Python SDK?

[NielsRogge]

Hi,

The Gemini Pro Vision and Gemini Pro 1.5 models look very performant from trying them out in Google AI Studio. However, the docs regarding API/Python SDK usage are pretty convoluted, hard to navigate through, and there's no guide on how to pass PDFs to Gemini.

What I could find is this:

• https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#gemini-pro-vision => showcases all HTTP parameters, but I assume there's also a Python SDK rather than having to use the requests library to pass all these parameters
• https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/send-multimodal-prompts => this page shows usage of the Python SDK, but only explains it for images and video. Is there a possibility to directly pass a PDF using the Python SDK?

[NielsRogge]

Looks like this is going to be addressed in #17 as the File API is still in preview

[NielsRogge]

Another thing that is confusing is that some guides are doing this:

!pip install -U -q google.generativeai

import google.generativeai as genai

model = genai.GenerativeModel('models/gemini-pro')

whereas other guides are using this:

!pip install --U -q google-cloud-aiplatform

from vertexai.generative_models import GenerativeModel

multimodal_model = GenerativeModel("gemini-1.0-pro-vision")

Which one is recommended? Why are there 2 Python SDKs?

[markmcd]

There are 2 SDKs because there are 2 platforms that host the Gemini API. One for Google Cloud Platform customers (minimal setup when running in Google Cloud), and one that does not require a GCP account (API key auth).

Platform	Docs	SDK
Gemini API	https://ai.google.dev/	google-generativeai
Vertex AI	https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview	google-cloud-aiplatform

This guide is meant to help disambiguate the two. We try to keep the API surfaces aligned, but some parts are different by design - e.g. authentication.

If you want to leave feedback for the docs you linked, you can use the "Send feedback" links on the respective pages.

[NielsRogge]

Ok, I've been trying out setting up an API key by following the authentication notebook.

However, Google AI Studio is not available in my region (Belgium). Does it mean I cannot use the Gemini API?

[edemir206]

@NielsRogge did find how to send PDF files to gemini api ? I'm struggling with this too, I want to send a PDF file via api for Gemini to summarize. Is it possible @markmcd ?

[NielsRogge]

Hi @edemir206,

here's the code one can use for that:

import vertexai
from vertexai.generative_models import GenerativeModel, Part

# variables for you to fill in
PROJECT_ID = ""
LOCATION = ""
BUCKET_NAME = ""

# initialize Vertex AI
vertexai.init(project=PROJECT_ID, location=LOCATION)

model = GenerativeModel("gemini-1.5-pro-preview-0409")

# get the GCS path of the file
gcs_path = f"gs://{BUCKET_NAME}/{filename}"

prompt = "Describe the file"

response = model.generate_content(
            [
                Part.from_uri(gcs_path, mime_type="application/pdf"),
                prompt
            ]
)

The code can be run if you're authenticated to your Google Cloud project (by running gcloud auth login).

[edemir206]

Hi @edemir206,

here's the code one can use for that:

import vertexai
from vertexai.generative_models import GenerativeModel, Part

# variables for you to fill in
PROJECT_ID = ""
LOCATION = ""
BUCKET_NAME = ""

# initialize Vertex AI
vertexai.init(project=PROJECT_ID, location=LOCATION)

model = GenerativeModel("gemini-1.5-pro-preview-0409")

# get the GCS path of the file
gcs_path = f"gs://{BUCKET_NAME}/{filename}"

prompt = "Describe the file"

response = model.generate_content(
            [
                Part.from_uri(gcs_path, mime_type="application/pdf"),
                prompt
            ]
)

The code can be run if you're authenticated to your Google Cloud project (by running gcloud auth login).

I'm using Google Gemini API not Vertex API, i'm still a little bit confused but I think for my needs Gemini Api Pricing is much lower.

I tried using inlineData for PDF embed with base64 encoded PDF but I get error 500, my json is formatted like:

{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Please summarize"
        },
        {
          "inlineData": {
            "mimeType": "application/pdf",
            "data": base64EncodedfileData
          }
        }
      ]
    }
  ],
  "generationConfig": {
    "temperature": 0.9,
    "topK": 1,
    "topP": 1,
    "maxOutputTokens": 2048,
    "stopSequences": []
  },
  "safetySettings": [
    {
      "category": "HARM_CATEGORY_HARASSMENT",
      "threshold": "BLOCK_MEDIUM_AND_ABOVE"
    },
    {
      "category": "HARM_CATEGORY_HATE_SPEECH",
      "threshold": "BLOCK_MEDIUM_AND_ABOVE"
    },
    {
      "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
      "threshold": "BLOCK_MEDIUM_AND_ABOVE"
    },
    {
      "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
      "threshold": "BLOCK_MEDIUM_AND_ABOVE"
    }
  ]
}

Is it not possible via Gemini API ?

[NielsRogge]

When you use Gemini's API you can follow the guide here: https://ai.google.dev/gemini-api/docs/prompting_with_media

[edemir206]

When you use Gemini's API you can follow the guide here: https://ai.google.dev/gemini-api/docs/prompting_with_media

I could upload the file using the sample but it fails with the error "google.api_core.exceptions.InvalidArgument: 400 Unsupported MIME type: application/pdf"

Here's my code:

import google.generativeai as genai
from IPython.display import Markdown

GOOGLE_API_KEY=""

genai.configure(api_key=GOOGLE_API_KEY)

sample_file = genai.upload_file(path="/home/user/python/test.pdf",
                                display_name="Sample PDF")

print(f"Uploaded file '{sample_file.display_name}' as: {sample_file.uri}")

model = genai.GenerativeModel(model_name="models/gemini-1.5-pro-latest")

response = model.generate_content(["Please Summarize PDF.", sample_file])

Markdown(">" + response.text)

genai.delete_file(sample_file.name)
print(f'Deleted {sample_file.display_name}.')