google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.

[zijianh4]

Description of the bug:

I met the error message google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable. frequently. Especially when I try to call the API multiple times using a loop. Is this a bug or is there a way to solve it? Thanks!

Actual vs expected behavior:

Should generate text or chat correctly but frequently have the error google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.

Any other information you'd like to share?

No response

[dinobby]

I am facing the same issue, and this did not happen a few days ago.

[ymodak]

Hi @zijianh4 and @dinobby, Thanks for reporting. Can you please share more information such as python version, region? Also do you see 503 error if you make api call once instead of multiple calls via loop? Thank you.

[zijianh4]

Hi @zijianh4 and @dinobby, Thanks for reporting. Can you please share more information such as python version, region? Also do you see 503 error if you make api call once instead of multiple calls via loop? Thank you.

Hi @ymodak , Sure. My Python version is 3.10.12 and region is US. I see 503 error when multiple calls via loop. I think it may be because the API is called too often so I tried to set some pause (~15s) between each call, but after some time period it still shows the 503 error. Thanks.

[mindwellsolutions]

SOLVED UPDATE: I was able to write code that automatically retried the query if the 503 error occurred and that fixed my issue even when the error appears. Code solution can be found in my last post Here.

Original Post: I wasn't getting this error yesterday but it started today. I've tried on multiple computers and multiple internet connections and with and without VPNs. All attempts including delaying queries by 1-5 seconds each, eventually run into the exact same "google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable." error.

Yesterday I ran a loop that had hundreds of queries in a row at much faster query rates and it worked without any issue at all.

[mindwellsolutions]

Update: All test queries were administered exactly the same way between all these events while making sure to stay under the 90 queries/responses per minute (Rate limit set by PaLM) using a script.

Now hours later its working perfectly again, I just processed 2,300 queries in a row without any 503 errors using the exact same process and datasets that produced the other errors at 10 queries and 70-200 queries before. So it's not the type of data/script being run with the PaLM API, because same data now works fine.

This leads me to believe its a problem on Google's PaLM API Servers or maybe some aspect of the PaLM API codebase that connects to Google PaLM.

Update: Got the 503 error again, but less frequently at the moment, although it's still happening - working on a better solution.

[mindwellsolutions]

SOLUTION that detects 503 errors, waits 2 seconds and then retries the last PaLM API query. This completely fixed my issue with the 503 error.

retry = True
while retry:
    try:
        completion = palm.generate_text(
        model=model,
        prompt=prompt1,
        temperature=0,
        # The maximum length of the response
        max_output_tokens=800,
    )
        retry = False
    except Exception as e:
        print(f"\033[1;31mAn exception occurred: {e}. Retrying in 2 seconds...\033[0m")
        time.sleep(2)

[MarkDaoust]

Hi, in recent releases we've added automatic retry logic to the underlying client library.

Like here: https://github.com/googleapis/google-cloud-python/blob/bd35749e76375874667b324d4b558a9bddc554c1/packages/google-ai-generativelanguage/google/ai/generativelanguage_v1/services/generative_service/transports/base.py#L129-L143

It has a 60s timeout for most methods, 600s for generate_content.

So I believe this is fixed.