Answered

[ferrenza]

Description of the bug:

I am experiencing intermittent 500 Internal Server Errors when using the gemini-1.5-pro model with the Google generative AI library. Sometimes the model responds correctly, but other times I encounter the error. This issue does not occur when using the gemini-1.5-flash model.

Sometimes gemini-1.5-pro can provide responses well, but other times it always returns: google.api_core.exceptions.InternalServerError: 500 An internal error has occurred. Please retry or report at https://developers.generativeai.google/guide/troubleshooting. The error occurs more often than the successful responses. I think Google needs to update the rate limit on their API or update this library to handle this issue when using gemini-1.5-pro.

Actual vs expected behavior:

Actual behavior: Intermittent 500 Internal Server Errors when using gemini-1.5-pro model. Expected behavior: Consistent and correct responses from the gemini-1.5-pro model without errors.

Any other information you'd like to share?

1. Use the gemini-1.5-pro model
2. Run the code and provide various inputs.
3. Example prompt: "hello world"
4. Answer: Responds normally.
5. On the second or third question, observe the 500 Internal Server Error:

Traceback (most recent call last): File "/root/ren/ai.py", line 30, in response = chat_session.send_message(user_input) File "/usr/local/lib/python3.10/dist-packages/google/generativeai/generative_models.py", line 578, in send_message response = self.model.generate_content( File "/usr/local/lib/python3.10/dist-packages/google/generativeai/generative_models.py", line 331, in generate_content response = self._client.generate_content( File "/usr/local/lib/python3.10/dist-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 827, in generate_content response = rpc( File "/usr/local/lib/python3.10/dist-packages/google/api_core/gapic_v1/method.py", line 131, in call return wrapped_func(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/usr/local/lib/python3.10/dist-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

This error appears intermittently when using gemini-1.5-pro but does not occur with gemini-1.5-flash.

[lc82111]

same issue from yesterday. and still persistent today.

[cauvang32]

I Got the same issue.

[justarmadillo]

Same issue !

[haicheviet]

I have had this problem before with only a few requests per day, but the current 500 rate is outrageous for making any production service from Gemini 1.5-pro.

[lc82111]

I'm in the free plan, are you in the pay plan?

I have had this problem before with only a few requests per day, but the current 500 rate is outrageous for making any production service from Gemini 1.5-pro.

[ferrenza]

I have had this problem before with only a few requests per day, but the current 500 rate is outrageous for making any production service from Gemini 1.5-pro.

I have tried creating an API key on a new Google account, and the result showed that even with a simple prompt like "hello world," it still resulted in a 500 Internal Server Error at that time.

[justarmadillo]

I have had this problem before with only a few requests per day, but the current 500 rate is outrageous for making any production service from Gemini 1.5-pro.

I have tried creating an API key on a new Google account, and the result showed that even with a simple prompt like "hello world," it still resulted in a 500 Internal Server Error at that time.

yes the same problem

[singhniraj08]

@ferrenza,

Thank you reporting this issue. This looks like an intermittent error and should work now. This repository is for issues related to Python SDK client bugs or improvements and the client doesn't control the service's responses. For issues related to Gemini API, we would suggest you to use "Send Feedback" option in Gemini docs. Ref: Screenshot below. You can also post this issue on Google AI forum.

[MarkDaoust]

Yeah, the service was overloaded. They are working on updating the 500 error to a 503, that "service unavailable" error should trigger automatic retries with backoff.

Also note that in addition to those automatic retries the "request_options" argument lets you pass a retry specification:

• https://github.com/google-gemini/generative-ai-python/blob/d0f3359a93a5192238bbf655ed57b6235913aef2/google/generativeai/generative_models.py#L237
• https://github.com/google-gemini/generative-ai-python/blob/d0f3359a93a5192238bbf655ed57b6235913aef2/google/generativeai/types/helper_types.py#L36

[htc7758521]

The same question, is it restricted?

[naarkhoo]

how to handle this error ?

[MarkDaoust]

I believe most 500's right now are because the service is overloaded. They are planning to upgrade those to 503 to make that clearer.

[SeaDude]

I'm getting 500 Internal error encountered when trying to cache ~1.8M tokens using gemini-1.5-pro-001.

2024-09-05T03:51:37Z   [Error]   #### Error details: {
    "message": "500 Internal error encountered.",
    "status_code": null,
    "traceback": [
        "Traceback (most recent call last):",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/google/api_core/grpc_helpers.py\", line 76, in error_remapped_callable",
        "    return callable_(*args, **kwargs)",
        "           ^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/grpc/_channel.py\", line 1181, in __call__",
        "    return _end_unary_response_blocking(state, call, False, None)",
        "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/grpc/_channel.py\", line 1006, in _end_unary_response_blocking",
        "    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable",
        "    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:",
        "\tstatus = StatusCode.INTERNAL",
        "\tdetails = \"Internal error encountered.\"",
        "\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:142.250.217.74:443 {grpc_message:\"Internal error encountered.\", grpc_status:13, created_time:\"2024-09-05T03:51:37.465556004+00:00\"}\"",
        ">",
        "",
        "The above exception was the direct cause of the following exception:",
        "",
        "Traceback (most recent call last):",
        "  File \"/home/site/wwwroot/shared/gemini_utils.py\", line 94, in create_new_gemini_cache",
        "    cached_content = caching.CachedContent.create(",
        "                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/vertexai/caching/_caching.py\", line 228, in create",
        "    cached_content_resource = client.create_cached_content(request)",
        "                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/google/cloud/aiplatform_v1beta1/services/gen_ai_cache_service/client.py\", line 825, in create_cached_content",
        "    response = rpc(",
        "               ^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/google/api_core/gapic_v1/method.py\", line 131, in __call__",
        "    return wrapped_func(*args, **kwargs)",
        "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
        "  File \"/home/site/wwwroot/.python_packages/lib/site-packages/google/api_core/grpc_helpers.py\", line 78, in error_remapped_callable",
        "    raise exceptions.from_grpc_error(exc) from exc",
        "google.api_core.exceptions.InternalServerError: 500 Internal error encountered."
    ]
}
2024-09-05T03:51:37Z   [Error]   ### Error: 500 Internal error encountered.

[MarkDaoust]

Interesting/ This is the first report I've seen of a 500 from caching.

Is it persistent? Only for giant contexts?

[naarkhoo]

I switched from "gemini-1.5-pro-latest" to "gemini-1.5-flash" and all those errors stopped - there were also some "out of quota" error;

[SeaDude]

I upgrade from vertexai v1.64.0 to 1.65.0 to fix...

[MarkDaoust]

I switched from "gemini-1.5-pro-latest" to "gemini-1.5-flash" and all those errors stopped - there were also some "out of quota" error;

Oh, yeah, that would do it. Quotas are much higher for flash.

[MarkDaoust]

I upgrade from vertexai v1.64.0 to 1.65.0 to fix...

I wouldn't expect changing local software to affect this (also this repo is not connected to vertex).

[naarkhoo]

@MarkDaoust I spent hours to figure out how to increase my quota and still don't know how ! Isn't all "pay as we go" ? Thank you.

[MarkDaoust]

https://ai.google.dev/pricing

No, you start on the free plan ... and (I think) if you setup billing for you project, then you're on the pay as you go plan.

[SeaDude]

I upgrade from vertexai v1.64.0 to 1.65.0 to fix...

I wouldn't ecpect changing local software to affect this (also this repo is not connected to vertex).

100% this was the issue. I spent a ridiculous amount of time tying everything. As soon as I upgraded the same code finally worked.