Gemini Batch Prediction API throws "INTERNAL" error

[afarahani-sharethis]

I am trying to test the batch prediction api for gemini with system instruction and contents. Below are the details for reproducing the example. After few minutes it ends up with the INTERNAL error with no further explanation. Not sure this is something from the google api side or there is a workaround. Thanks for your help.

Environment details

• OS type and version: macOS 13.6 (22G120)
• Python version: 3.10.4
• pip version: 23.2
• google-cloud-aiplatform version: 1.57.0

Steps to reproduce

1. The input dataset is a jsonl file sample as follows:

{"url": "url1", "request": {"contents": [{"role": "user", "parts": [{"text": "Some of these beautiful gems hide their light under a dry bushel in drought years, so when the (rare) precipitation pours into Los Angeles, step into nature and admire the impressive flowing waters. But don't forget an extra pair of socks when embarking on waterfall hikes near Los Angeles because your feet might get wet from those creek crossings. Escondido Falls is one of SoCals most popular and dramatic waterfalls. The first .75 mile of this hike is along apaved road in front of expensive Malibu homes, but the trail turns more rustic once you join the dirt road marked Edward Albert Escondido Canyon Trail and Waterfall. Be prepared to get your feet wet while crossing streams along the route. You can only access the lower falls as the upper falls are closed to the public. To get to the trailhead: Take PCH to Winding Way in Malibu. Winding Way is a minor street 4.5 miles west of Malibu Canyon Road and 1.4 miles east of Kanan Dume Road on the north side of PCH. Turn up Winding Way into a paid parking lot for the trail on the left at the bottom of Winding Way. Parking in this lot is pay-and-display. RELATED: Hidden Gem: Ferndell is an Outdoor Oasis in the Middle of the City On the Solstice Canyon Trail Loop. Photo by Jeremy Miles via Flickr"}]}], "system_instruction": {"parts": [{"text": "You are an assistant that extracts specific information from texts. You receive a text and respond with the extracted information in JSON format. The information you need to extract includes categories (IAB). Your response must be a JSON object containing the extracted information. The JSON object has the following schema: categories: A list of IAB Content Categories form Internet Advertising Bureau found in the text"}]}}} {"url": "url2", "request": {"contents": [{"role": "user", "parts": [{"text": "The new musical Midnight in the Garden of Good and Evil\u2014a world-premiere musical based on John Berendt\u2019s iconic non-fiction book is gearing up for its world premiere at Chicago's Goodman Theatre. Hear the show's choreographer Tanya Birl-Torres discuss how the dirt of Savannah informed the process of making Midnight in the Garden of Good and Evil dance in the video. Hear from the show's Tony-winning director, Rob Ashford here. Midnight in the Garden of Good and Evil\u2014John Berendt\u2019s 1994 blockbuster non-fiction book, a Pulitzer-Prize finalist that was on The New York Times Best-Seller list for 216 weeks\u2014becomes a seductive new musical. Southern charm is bountiful in Savannah, Georgia. But behind polite smiles, the eccentric residents are filled with secrets and motives. When wealthy antiques dealer Jim Williams is accused of murder, the sensational trial uncovers hidden truths and exposes the fine line between good and evil\u2014which sparks Lady Chablis and other Savannahians to change the city forever."}]}], "system_instruction": {"parts": [{"text": "You are an assistant that extracts specific information from texts. You receive a text and respond with the extracted information in JSON format. The information you need to extract includes categories (IAB). Your response must be a JSON object containing the extracted information. The JSON object has the following schema: categories: A list of IAB Content Categories form Internet Advertising Bureau found in the text"}]}}}

2. run this code after installing the required packages.

   
   import time
   import vertexai
   from vertexai.preview.batch_prediction import BatchPredictionJob

project_id = "the_project_id" location = "the_location" staging_bucket = "gs://bucket_name"

vertexai.init(project=project_id, location=location, staging_bucket=staging_bucket)

job = BatchPredictionJob.submit( source_model="gemini-1.0-pro", # source_model input_dataset="gs://path/to/input/dataset.jsonl", output_uri_prefix="gs://path/to/output/uri/prefix", )

Check job status

print(f"Job resouce name: {job.resource_name}") print(f"Model resource name with the job: {job.model_name}") print(f"Job state: {job.state.name}")

Refresh the job until complete

while not job.has_ended: time.sleep(5) job.refresh()

Check if the job succeeds

if job.has_succeeded: print("Job succeeded!") else: print(f"Job failed: {job.error}")

Check the location of the output

print(f"Job output location: {job.output_location}")

List all the GenAI batch prediction jobs under the project

for bpj in BatchPredictionJob.list(): print( f"Job ID: '{bpj.name}', Job state: {bpj.state.name}, Job model: {bpj.model_name}" )

#### Printed Error Detail

Model resource name with the job: publishers/google/models/gemini-1.0-pro Job state: JOB_STATE_PENDING Job failed: code: 13 message: "INTERNAL"

[jaycee-li]

Hi @afarahani-sharethis, thanks for bringing this up! I'm able to reproduce the issue. It seems the service API currently only accepts plain text for GCS input data (e.g., {"request": "Give me a recipe for banana bread."}). I'll file a ticket to the backend team.

As a workaround, you could try using BigQuery as your input data source. This would allow you to use system instructions.

You can load you gcs file into bigquery like this

from google.cloud import bigquery

client = bigquery.Client()

job_config = bigquery.LoadJobConfig(
    schema=[
        bigquery.SchemaField("url", "STRING"),
        bigquery.SchemaField("request", "JSON"),
    ],
    source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
)
load_job = client.load_table_from_uri(
    source_uris="gs://path/to/input/dataset.jsonl",
    destination="your_project.your_dataset.your_table",
    job_config=job_config,
)

load_job.result()

and then run the batch prediction with bq input

job = BatchPredictionJob.submit(
    source_model="gemini-1.0-pro", 
    input_dataset="your_project.your_dataset.your_table",
    output_uri_prefix="gs://path/to/output/uri/prefix",
)

[afarahani-sharethis]

I was able to fix the error by your BigQuery workaround. thank you so much. One minor correction to your code. the bq table url acceptable format is bq://projectId.bqDatasetId.bqTableId.

[afarahani-sharethis]

I have another question regarding the gemini batch api. can one use all the possible parameters in the batch request such as tools(grounding), generation_config and so on? if this is not related, I can test and open another issue in case it throws another error.

[jaycee-li]

Yes, all the parameters should work in batch prediction.