Consistent 500 error

[ianb]

Description of the bug:

I get lots of intermittent 500 errors (lots and lots!) but this request gives consistent 500 errors:

{"contents":
  [
    {"role":"user","parts":[{"text":""},{"text":""}]}
  ],
  "tools":[{"functionDeclarations":[{"name":"locationFamilyInfo","description":"Identify the status/classes of people in Mycenae in 1200BC.","parameters":{"type":"object","properties":{"classes":{"type":"array","items":{"type":"string","description":"A list of classes or status of people in the location, such as \"upper class\", \"surf\", \"untouchable\", etc. Include some exceptional classes, such as \"royalty\", \"nobility\", \"priesthood\", \"warrior\", etc. Give the full range of classes/status in society at the time."}},"ethnicities":{"type":"array","items":{"type":"string","description":"A list of races and ethnicities in the location, such as \"white\", \"hispanic\", \"Hmong\", etc."}},"religions":{"type":"array","items":{"type":"string","description":"A list of religions in the location, such as \"Christian\", \"Muslim\", \"Hindu\", etc. Can include denominations, such as \"Catholic\", \"Protestant\", \"Sunni\", \"Shia\", etc."}}},"required":["classes","ethnicities","religions"]}}]}],
  "generationConfig":{"temperature":0.5}
}

Notably the request text is weird. GPT answers this successfully and accurately based on the function call alone.

Actual vs expected behavior:

No 500 error

Any other information you'd like to share?

No response

[kpripper]

I have the same problem. Approximately half of the requests give the error 500. The frequency of requests is 2 per minute. No more than 1,000 tokens per prompt.

[davidstackio]

In the last few days I've also been getting more 500 errors as well (beyond the usual SAFETY errors). I'm using the JavaScript SDK in Node v18.19.0 and Nuxt v3.9.0.

My error is:

GoogleGenerativeAIError: [500 Internal Server Error] An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting
    at makeRequest (file:///workspace/node_modules/@google/generative-ai/dist/index.mjs:214:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateContent (file:///workspace/node_modules/@google/generative-ai/dist/index.mjs:523:22)
    at async ChatSession.sendMessage (file:///workspace/node_modules/@google/generative-ai/dist/index.mjs:667:9)
    at async file:///workspace/chunks/chat.post.mjs:136:22
    at async Object.handler (file:///workspace/chunks/nitro/firebase-gen-2.mjs:2383:19)
    at async toNodeHandle (file:///workspace/chunks/nitro/firebase-gen-2.mjs:2572:7)

Which brought me here.

This is my model config:

const genAI = new GoogleGenerativeAI(process.env.GOOGLE_AI_API_KEY as string);
const model = genAI.getGenerativeModel({
  model: "gemini-pro",
  generationConfig: {
    maxOutputTokens: 1000,
    temperature: 0.8,
    topK: 3,
    topP: 0.95,
  },
});

[kpripper]

Do you use simultaneous requests with different API keys from one IP address?

[davidstackio]

@kpripper I do not. I just have one API key from one domain.

[MarkDaoust]

Hi, everyone. Yes, the service was having capacity issues recently. This should be running better now.

[keertk]

Closing this issue since it's been resolved, but please feel free to reopen or raise a new issue if you see this again.

[shaayaansh]

I'm getting this error in the past week. I'm using colab and just one API. This is how I initialized my model:

safety_settings = [{
  "category": "HARM_CATEGORY_HARASSMENT",
  "threshold": "HIGH"
},
{"category": "HARM_CATEGORY_HATE_SPEECH",
 "threshold": "BLOCK_NONE"}]

model = genai.GenerativeModel('gemini-pro', safety_settings=safety_settings)

I'm running the following code in a for loop and none of my inputs exceed 1000 tokens. The error occurs sometimes after 150 iterations and some times after the first 10. I don't understand whats happening.

   response = model.generate_content(prompt.format(sent),
                                      generation_config=genai.types.GenerationConfig(
                                          candidate_count=1, max_output_tokens=10,
                                          temperature=1.0))

[1andDone]

Seeing the same error on my end. Is the solution to just wrap the request in a retry?

[deathemperor]

Is the capacity issue happening again? Been getting 500 for all request for more than 24 hours now

[MarkDaoust]

I don't see all the internal struggles. It could be that, just be sure to try a vanilla request to rule out some bad input causing it to fail.

[deathemperor]

I don't see all the internal struggles. It could be that, just be sure to try a vanilla request to rule out some bad input causing it to fail.

it is a vanilla request taken from https://cloud.google.com/nodejs/docs/reference/vertexai/latest

the sample source

const {
  FunctionDeclarationSchemaType,
  HarmBlockThreshold,
  HarmCategory,
  VertexAI,
} = require("@google-cloud/vertexai");

const project = "careportal-357509";
const location = "us-central1";
const textModel = "gemini-1.0-pro";
const visionModel = "gemini-1.0-pro-vision";

const vertexAI = new VertexAI({ project: project, location: location });

// Instantiate Gemini models
const generativeModel = vertexAI.getGenerativeModel({
  model: textModel,
  // The following parameters are optional
  // They can also be passed to individual content generation requests
  safetySettings: [
    {
      category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
      threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,
    },
  ],
  generationConfig: {
    maxOutputTokens: 256,
    responseMimeType: "application/json",
  },
});

const generativeVisionModel = vertexAI.getGenerativeModel({
  model: visionModel,
});

const generativeModelPreview = vertexAI.preview.getGenerativeModel({
  model: textModel,
});

async function generateContent() {
  const request = {
    contents: [{ role: "user", parts: [{ text: "How are you doing today?" }] }],
  };
  const result = await generativeModel.generateContent(request);
  const response = result.response;
  console.log("Response: ", JSON.stringify(response));
}

generateContent();

[deathemperor]

Just tried to make it really vanilla by removing responseMimeType: "application/json", and it works. It's not like this 5 days ago.

I know responseMimeType: "application/json", is not supported in JS sdk but still tried it and it worked which helped my application. Now suddenly it doesn't and my application stopped working.

[deathemperor]

Issue mentioned at https://github.com/googleapis/nodejs-vertexai/issues/306

[MarkDaoust]

wrap the request in a retry?

Also note that generate_content accepts a request_options = {'retry': google.api_core.retry.Retry(...)} argument. That can be configured to automatically retry failures a number of times.

[MarkDaoust]

For @ianb

Yeah, it looks like it fails if there's no input:

model.generate_content(
  contents=[{"role":"user","parts":[{"text":""}]}],
)

[MarkDaoust]

@deathemperor

Errors may be different between us and vertex. Are you able to reproduce this with google.generativeai? This seems to work fine:

model = genai.GenerativeModel('gemini-1.5-pro-latest')

model.generate_content(
  contents="How are you doing today?",
  generation_config={'response_mime_type':'application/json'}
)

response.text #  '{\n\n"response": "I am an AI language model, so I do not have feelings or experiences like humans do. However, I am here to assist you with any questions or tasks you may have." \n\n}\n\n'

[deathemperor]

@MarkDaoust I understand the config is supported in python but I'm using nodejs sdk, tried with gemini-1.5-pro-preview-0409 (gemini-1.5-pro-latest throws Publisher Model not found for me) doesn't work.

[deathemperor]

After submitting a support ticket with GCP, I checked again today and the 500 error is gone.