search

googleapis / python-bigquery

Apache License 2.0
746 stars 306 forks source link

No retry on Impersonation Token Bigquery #2013

Open nitishxp opened 2 months ago

nitishxp commented 2 months ago

Hi team,

Using google-cloud-bigquery==3.24.0 on python 3.11 cloud function runtime

File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 494, in api_request return self._call_api( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 833, in _call_api return call() 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 482, in api_request response = self._make_request( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request return self._do_request( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request return self.http.request( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/transport/requests.py", line 534, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/credentials.py", line 230, in before_request self._blocking_refresh(request) File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/credentials.py", line 193, in _blocking_refresh self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/impersonated_credentials.py", line 235, in refresh self._update_token(request) File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/impersonated_credentials.py", line 267, in _update_token self.token, self.expiry = _make_iam_token_request( 
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/auth/impersonated_credentials.py", line 83, in _make_iam_token_request raise exceptions.RefreshError(_REFRESH_ERROR, response_body) google.auth.exceptions.RefreshError: ('Unable to acquire impersonated credentials', '<!DOCTYPE html>\n<html lang=en>\n <meta charset=utf-8>\n <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">\n <title>Error 502 (Server Error)!!1</title>\n <style>\n *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n </style>\n <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n <p><b>502.</b> <ins>That’s an error.</ins>\n <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>\n')
shollyman commented 2 months ago

I'm not super clear on the flow here, but there's a reliance is on the google-auth package which provides it's own retry mechanisms within the package. If the concern is about retries within the auth library, could you provide those details to that project at https://github.com/googleapis/google-auth-library-python?

alfonsrv commented 1 month ago

There's some kind of change or global error in the Google Cloud, given this error + HTML returned from the API endpoints is fairly new. I had an automation running for 4 years+ now and this is an issue that just recently came up, since it breaks JSON decoding.

shollyman commented 1 month ago

Adding @chalmerlowe here, as I'll be unavailable to investigate further.