search

google-gemini / generative-ai-python

The official Python library for the Google Gemini API
https://pypi.org/project/google-generativeai/
Apache License 2.0
1.62k stars 322 forks source link

Internal Server Error (500) when using Gemini API with Inspect framework #573

Open lennijusten opened 1 month ago

lennijusten commented 1 month ago

Description of the bug:

Description

I'm consistently encountering an Internal Server Error (HTTP 500) when trying to use the Google Gemini API through the Inspect evaluation framework. This error occurs during the generate_content call. I'm currently on the free trial.

Steps to Reproduce

  1. Set up an evaluation using the Inspect framework
  2. Configure the evaluation to use the Gemini model (in my case, google/gemini-1.5-pro)
  3. Run the evaluation

Error Message

InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

Environment

Package Versions

Full error traceback:

╭─ benchmarks/gpqa (78 samples): google/gemini-1.5-pro ─────────────────────────────────────────────────────────────────────╮
│ ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮   dataset: (samples) │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │       scorer: choice │
│ │ in task_run                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_run_sample                                                                               │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in solve                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in task_generate                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in _generate                                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in async_wrapped                                                                                 │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in iter                                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/_u… │                      │
│ │ in inner                                                                                         │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/__… │                      │
│ │ in <lambda>                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in result                                                                                        │                      │
│ │                                                                                                  │                      │
│ │   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                     │                      │
│ │   447 │   │   │   │   │   raise CancelledError()                                                 │                      │
│ │   448 │   │   │   │   elif self._state == FINISHED:                                              │                      │
│ │ ❱ 449 │   │   │   │   │   return self.__get_result()                                             │                      │
│ │   450 │   │   │   │                                                                              │                      │
│ │   451 │   │   │   │   self._condition.wait(timeout)                                              │                      │
│ │   452                                                                                            │                      │
│ │                                                                                                  │                      │
│ │ /opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.1… │                      │
│ │ in __get_result                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   398 │   def __get_result(self):                                                                │                      │
│ │   399 │   │   if self._exception:                                                                │                      │
│ │   400 │   │   │   try:                                                                           │                      │
│ │ ❱ 401 │   │   │   │   raise self._exception                                                      │                      │
│ │   402 │   │   │   finally:                                                                       │                      │
│ │   403 │   │   │   │   # Break a reference cycle with the exception in self._exception            │                      │
│ │   404 │   │   │   │   self = None                                                                │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/tenacity/as… │                      │
│ │ in __call__                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/inspect_ai/… │                      │
│ │ in generate                                                                                      │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/gene… │                      │
│ │ in generate_content_async                                                                        │                      │
│ │                                                                                                  │                      │
│ │   382 │   │   │   │   │   )                                                                      │                      │
│ │   383 │   │   │   │   return await generation_types.AsyncGenerateContentResponse.from_aiterato   │                      │
│ │   384 │   │   │   else:                                                                          │                      │
│ │ ❱ 385 │   │   │   │   response = await self._async_client.generate_content(                      │                      │
│ │   386 │   │   │   │   │   request,                                                               │                      │
│ │   387 │   │   │   │   │   **request_options,                                                     │                      │
│ │   388 │   │   │   │   )                                                                          │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/ai/g… │                      │
│ │ in generate_content                                                                              │                      │
│ │                                                                                                  │                      │
│ │    403 │   │   self._client._validate_universe_domain()                                          │                      │
│ │    404 │   │                                                                                     │                      │
│ │    405 │   │   # Send the request.                                                               │                      │
│ │ ❱  406 │   │   response = await rpc(                                                             │                      │
│ │    407 │   │   │   request,                                                                      │                      │
│ │    408 │   │   │   retry=retry,                                                                  │                      │
│ │    409 │   │   │   timeout=timeout,                                                              │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_wrapped_func                                                                            │                      │
│ │                                                                                                  │                      │
│ │   227 │   │   │   sleep_generator = exponential_sleep_generator(                                 │                      │
│ │   228 │   │   │   │   self._initial, self._maximum, multiplier=self._multiplier                  │                      │
│ │   229 │   │   │   )                                                                              │                      │
│ │ ❱ 230 │   │   │   return await retry_target(                                                     │                      │
│ │   231 │   │   │   │   functools.partial(func, *args, **kwargs),                                  │                      │
│ │   232 │   │   │   │   predicate=self._predicate,                                                 │                      │
│ │   233 │   │   │   │   sleep_generator=sleep_generator,                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │   159 │   │   │   # defer to shared logic for handling errors                                    │                      │
│ │ ❱ 160 │   │   │   _retry_error_helper(                                                           │                      │
│ │   161 │   │   │   │   exc,                                                                       │                      │
│ │   162 │   │   │   │   deadline,                                                                  │                      │
│ │   163 │   │   │   │   sleep,                                                                     │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in _retry_error_helper                                                                           │                      │
│ │                                                                                                  │                      │
│ │   209 │   │   │   RetryFailureReason.NON_RETRYABLE_ERROR,                                        │                      │
│ │   210 │   │   │   original_timeout,                                                              │                      │
│ │   211 │   │   )                                                                                  │                      │
│ │ ❱ 212 │   │   raise final_exc from source_exc                                                    │                      │
│ │   213 │   if on_error_fn is not None:                                                            │                      │
│ │   214 │   │   on_error_fn(exc)                                                                   │                      │
│ │   215 │   if deadline is not None and time.monotonic() + next_sleep > deadline:                  │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in retry_target                                                                                  │                      │
│ │                                                                                                  │                      │
│ │   152 │                                                                                          │                      │
│ │   153 │   for sleep in sleep_generator:                                                          │                      │
│ │   154 │   │   try:                                                                               │                      │
│ │ ❱ 155 │   │   │   return await target()                                                          │                      │
│ │   156 │   │   # pylint: disable=broad-except                                                     │                      │
│ │   157 │   │   # This function explicitly must deal with broad exceptions.                        │                      │
│ │   158 │   │   except Exception as exc:                                                           │                      │
│ │                                                                                                  │                      │
│ │ /Users/lenni/Documents/GitHub/biology-benchmarks/.venv/lib/python3.12/site-packages/google/api_… │                      │
│ │ in __await__                                                                                     │                      │
│ │                                                                                                  │                      │
│ │    85 │   │   │   response = yield from self._call.__await__()                                   │                      │
│ │    86 │   │   │   return response                                                                │                      │
│ │    87 │   │   except grpc.RpcError as rpc_error:                                                 │                      │
│ │ ❱  88 │   │   │   raise exceptions.from_grpc_error(rpc_error) from rpc_error                     │                      │
│ │    89                                                                                            │                      │
│ │    90                                                                                            │                      │
│ │    91 class _WrappedStreamResponseMixin(Generic[P], _WrappedCall):                               │                      │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                      │
│ InternalServerError: 500 An internal error has occurred. Please retry or report in                                        │
│ https://developers.generativeai.google/guide/troubleshooting

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

gmKeshari commented 1 month ago

Hi @lennijusten ,

Have you been facing this issue since the beginning ? I mean is it like you were getting the response earlier and now facing this issue?

Try upgrading the google.generative library once and then run your commands :

pip install -q --upgrade google-generativeai

Please let us know if you are still having the issue.

lennijusten commented 1 month ago

Yes. It's been a semi-persistent problem throughout my usage of Gemini. Sometimes it works, but then after some number of requests or tokens, I get the Internal Server Error (500). The issue is also discussed here and here on Reddit.

I ran pip install -q --upgrade google-generativeai which upgraded me from google-generativeai==0.6.6 to google-generativeai==0.8.2 but the issue persists.

If there is some kind of rate limit happening it would be useful for the error code to reflect that.

With the current workings, I'm just burning tokens for nothing since my task never completes before the error.

gmKeshari commented 1 month ago

Hi @lennijusten . Sometimes we see Internal Server Error (500) because there are too many requests coming, which results in an unexpected error on Google's side. You can refer to this Troubleshooting doc "https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python". I recommend temporarily switching to another model (e.g. from Gemini 1.5 Pro to Gemini 1.5 Flash) and see if it works.

Internal Server Error (500) is not related to rate limit, but free tier does have rate limits based model variation. Please refer to this doc : "https://ai.google.dev/gemini-api/docs/models/gemini" or "https://ai.google.dev/pricing"

If you want to upgrade to "Pay-as-you-go" tier, you need to set up a billing account.