`socket.timeout: timed out` when using full IPv6 stack.

n8felton commented 5 years ago

Environment details

OS: macOS 10.14.5 (18F132) Mojave
Python version: 3.7.3
pip version: 19.1.1
google-api-python-client version: 1.7.9

Steps to reproduce

Pre-steps:

Have a full IPv6 stack with an IPv6 address. In my case, I have the following from Spectrum, through a Linksys WRT3200ACM router.

inet6 fe80::4ab:197d:b439:8d97%en0 prefixlen 64 secured scopeid 0x9 
inet6 2604:6000:1013:a0eb:cca:87bb:d7ed:91b6 prefixlen 64 autoconf secured 
inet6 2604:6000:1013:a0eb:91e1:1128:7728:8e1 prefixlen 64 autoconf temporary

Error:

Follow the steps for the Python Quickstart (https://developers.google.com/docs/api/quickstart/python)
run python quickstart.py
python exits with socket.timeout: timed out error

If I open System Preferences > Network > Wi-FI > Advanced > TCP/IP and change the IPv6 configuration from Automatically to Link-local only, the quickstart.py properly runs.

busunkim96 commented 5 years ago

A few folks have filed issues about socket timeouts recently - thank you for the additional info!

vladignatyev commented 4 years ago

Socket problem is still here, in 2020.

Environment details

OS: Alpine (docker image https://hub.docker.com/_/python python:3.8.0-alpine)

Container runs on AWS EC2 instance.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/src/app/searchconsole/data.py", line 90, in load_data_for_property
    save_data_batch(property, batch)
  File "/usr/src/app/searchconsole/data.py", line 103, in save_data_batch
    for data_batch in data_batches:
  File "/usr/src/app/searchconsole/google/reports.py", line 103, in generate_lazy
    response = request.execute()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/google_auth_httplib2.py", line 197, in request
    response, content = self.http.request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
    (response, content) = self._request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
    (response, content) = self._conn_request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

gselva commented 4 years ago

@vladignatyev - I had this error and was able to get around it by updating /etc/gai.conf to prefer IPV4 over IPV6 (uncomment the line precedence ::ffff:0:0/96 100 ).

cheptsov commented 4 years ago

I have this issue too

tuliocgon commented 4 years ago

I'm experiencing this right now, the whole weekend, a code that was previoudly fine now i can't get pass authentication:

.../python3.8/site-packages/httplib2/init.py, line 1324, in connect sock.connect((self.host.self.port__ socket.timetou: time out

Any advices how to solve this ?

vlad-oles commented 4 years ago

I also started experiencing this issue just a day ago, with no modification to the code:

    service = build('sheets', 'v4', credentials=get_creds())
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 225, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 282, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request
    conn.connect()
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect
    sock.connect((self.host, self.port))

OS: Ubuntu 18.04.5 LTS
Python: 3.7.6
pip: 20.0.2
googleapiclient: 1.8.0

Uncommenting the line precedence ::ffff:0:0/96 100 in /etc/gai.conf fixed the problem.

SurferJeffAtGoogle commented 4 years ago

My first attempt to reproduce on a Windows machine did not manifest the bug. Trying Linux....

SurferJeffAtGoogle commented 4 years ago

Tried on Linux, haven't manifested the bug yet. Still working on it...

SurferJeffAtGoogle commented 4 years ago

How often do you see the socket timeout? With every request? What fraction of your requests?

SurferJeffAtGoogle commented 4 years ago

Has your code called socket.setdefaulttimeout(timeout) before calling this function?

What value is returned by socket.getdefaulttimeout() immediately before calling the API?

SurferJeffAtGoogle commented 4 years ago

After studying the code in the client library, I don't see any red flags in the client library itself. An n8felton's stack trace demonstrates, the google client calls httplib2, which calls socket functions, where the timeout happens.

SurferJeffAtGoogle commented 4 years ago

I'm trying to reproduce this issue on Alpine Linux running on an EC2 instance. When I run the quickstart (https://developers.google.com/docs/api/quickstart/python), it asks me to visit a localhost website to authenticate, which I can't do, since I'm connected via ssh with no graphical user interface.

Does someone have a simpler way to reproduce the misbehavior?

cfiner commented 4 years ago

I'm having the same issue. Code was running fine a month ago, tried today and I get a timeout error.

[---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
<ipython-input-44-75c2f3716daf> in <module>()
----> 1 service = build('calendar','v3', credentials=credentials)

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py in positional_wrapper(*args, **kwargs)
    132                 elif positional_parameters_enforcement == POSITIONAL_WARNING:
    133                     logger.warning(message)
--> 134             return wrapped(*args, **kwargs)
    135 
    136         return positional_wrapper

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py in build(serviceName, version, http, discoveryServiceUrl, developerKey, model, requestBuilder, credentials, cache_discovery, cache, client_options)
    223       options to set user options on the client.
    224       (1) The API endpoint should be set through client_options. If API endpoint
--> 225       is not set, `GOOGLE_API_USE_MTLS_ENDPOINT` environment variable can be used
    226       to control which endpoint to use.
    227       (2) client_cert_source is not supported, client cert should be provided using

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py in _retrieve_discovery_doc(url, http, cache_discovery, cache, developerKey)
    280                 base=discovery_url,
    281                 http=http,
--> 282                 developerKey=developerKey,
    283                 model=model,
    284                 requestBuilder=requestBuilder,

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in request(self, uri, method, body, headers, redirections, connection_type)
   1974                         headers,
   1975                         redirections,
-> 1976                         cachekey,
   1977                     )
   1978         except Exception as e:

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in _request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
   1638 
   1639         (response, content) = self._conn_request(
-> 1640             conn, request_uri, method, body, headers
   1641         )
   1642 

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in _conn_request(self, conn, request_uri, method, body, headers)
   1544             try:
   1545                 if conn.sock is None:
-> 1546                     conn.connect()
   1547                 conn.request(method, request_uri, body, headers)
   1548             except socket.timeout:

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in connect(self)
   1315                 if has_timeout(self.timeout):
   1316                     sock.settimeout(self.timeout)
-> 1317                 sock.connect((self.host, self.port))
   1318 
   1319                 self.sock = self._context.wrap_socket(sock, server_hostname=self.host)

timeout: timed out](url)

max-sixty commented 4 years ago

If another case is helpful, I'm getting the same error when attempting to instantiate a storagetransfer client, running from GKE. From my laptop, using the same docker image, I can't repro.

   File "/usr/local/lib/python3.8/site-packages/{...}/load.py", line 137, in _storage_transfer_client
     return googleapiclient.discovery.build("storagetransfer", "v1", cache=MemoryCache())
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
     return wrapped(*args, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 245, in build
     content = _retrieve_discovery_doc(
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 305, in _retrieve_discovery_doc
     resp, content = http.request(actual_url)
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1985, in request
     (response, content) = self._request(
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
     (response, content) = self._conn_request(
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
     response = conn.getresponse()
   File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
     response.begin()
   File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
     version, status, reason = self._read_status()
   File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
   File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
     return self._sock.recv_into(b)
   File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
     return self.read(nbytes, buffer)
   File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
     return self._sslobj.read(len, buffer)
 socket.timeout: The read operation timed out

max-sixty commented 4 years ago

My problem was solved by duplicating my GKE cluster, and it working on the new GKE cluster. So not sure what the actual problem was — calling build with any valid service python -c 'import googleapiclient.discovery; googleapiclient.discovery.build("compute", "v1")' would hang before timing out.

SurferJeffAtGoogle commented 4 years ago

Thank you @max-sixty, that's a really useful data point. I will follow up with GKE engineers. Is the old, malfunctioning cluster still running?

SurferJeffAtGoogle commented 4 years ago

I filed internal tracking bug 171068451.

max-sixty commented 4 years ago

Hi @SurferJeffAtGoogle — this ended up being a GKE issue, and the "it works on another cluster" was because the problem hadn't spread to another zone at that point; nothing to do with the cluster itself (it later spread to all zones).

I can't see it in the Incident History, but it was listed under https://console.cloud.google.com/support/cases;tab=knownissues for a while. Our Google support case number was 25392673. Given it spanned half a dozen regions, I would have expected to see it in the Incident History.

Let me know any questions! This was fairly painful for us — seemingly more so than for others, I still haven't fully resolved why.

hkdevandla commented 4 years ago

@max-sixty , can you tell us if you are still seeing the timeout issue? There was an issue related to API discovery service between 2020-10-06 to 2020-10-08 from GCE VMs which may have caused this timeout issue as well and the changes were rolled back to mitigate the issue. So want to check if it is still happening. Please advice.

max-sixty commented 4 years ago

This was fixed with the rollback and is no longer an issue for us, I should have made that clearer above.

There was an issue related to API discovery service between 2020-10-06 to 2020-10-08 from GCE VMs which may have caused this timeout issue as well and the changes were rolled back to mitigate the issue.

I can't see this on the incident history, I'm guessing you see this internally

SurferJeffAtGoogle commented 4 years ago

There have been recent changes to our network configuration. Would someone who has experienced this issue please confirm it's still happening?

meeranagpal commented 4 years ago

I just started getting this issue 2 days ago. I'm using the Google Custom Search API. Does anyone know how I can solve this?

Traceback (most recent call last): File "serp_googlecs.py", line 73, in <module> format_search_results('75_dataset.csv') File "serp_googlecs.py", line 56, in format_search_results queries, names = combined_serp_cs(data) File "serp_googlecs.py", line 50, in combined_serp_cs res = google_api_search(query, my_api_key, my_cse_id) File "serp_googlecs.py", line 32, in google_api_search res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute() File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper return wrapped(*args, **kwargs) File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 901, in execute headers=self.headers, File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 204, in _retry_request raise exception File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 177, in _retry_request resp, content = http.request(uri, method, *args, **kwargs) File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request cachekey, File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request conn, request_uri, method, body, headers File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request conn.connect() File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect sock.connect((self.host, self.port)) socket.timeout: timed out

DanielAviv commented 4 years ago

I just got it, previously working code now does not pass the auth.

File "lib/python3.8/site-packages/httplib2/__init__.py", line 1324, in connect sock.connect((self.host, self.port)) socket.timeout: timed out

Any fix for this?

Edit: Uncommenting the line worked for me on Ubuntu 20

I also started experiencing this issue just a day ago, with no modification to the code:

    service = build('sheets', 'v4', credentials=get_creds())
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 225, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 282, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request
    conn.connect()
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect
    sock.connect((self.host, self.port))

OS: Ubuntu 18.04.5 LTS
Python: 3.7.6
pip: 20.0.2
googleapiclient: 1.8.0

Uncommenting the line precedence ::ffff:0:0/96 100 in /etc/gai.conf fixed the problem.

SurferJeffAtGoogle commented 3 years ago

@parthea If you're seeking a very challenging bug to investigate, try this one!

parthea commented 3 years ago

@nulld mentioned a workaround in #563 where he had to move the socket.setdefaulttimeout() function before import googleapiclient. If it solves the problem, then credit goes to @nulld ! Either way, we can use this data point to better understand whether there are multiple issues going on. Please can someone who is experiencing the issue try the workaround and report back?

busunkim96 commented 3 years ago

Reassigning to @hkdevandla. Googlers see status in 171349628

parthea commented 3 years ago

Remove priority:p2 label based on internal status b/171349628

chrissachs commented 3 years ago

I think I see the same issue, but it's not related to ipv6, or the python library.

I found this curl to reproduce it:

curl --http1.1 -v --request GET 'https://analytics.googleapis.com/analytics/v3/metadata/ga/columns?alt=json' --header 'Authorization: Bearer ya29.... -v

It usually works without problems, but about every 3rd or 4th request gets "stuck" and never finishes.

I'm running this command inside of a container based on python:3.6-slim, inside of a kubernetes cluster with nodes in europe-west-3. It's the same with http1.1 and http2

The logs of a successfull request look like this:

curl -4 --http2 --request GET 'https://analytics.googleapis.com/analytics/v3/metadata/ga/columns?alt=json' --header 'Authorization: Bearer ya29....' -v > /dev/null
Note: Unnecessary use of -X or --request, GET is already inferred.
* Expire in 0 ms for 6 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 0 ms for 1 (transfer 0x55560bd5af90)
* Expire in 0 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 0 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 1 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 4 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 2 ms for 1 (transfer 0x55560bd5af90)
* Expire in 4 ms for 1 (transfer 0x55560bd5af90)
* Expire in 3 ms for 1 (transfer 0x55560bd5af90)
* Expire in 3 ms for 1 (transfer 0x55560bd5af90)
* Expire in 4 ms for 1 (transfer 0x55560bd5af90)
* Expire in 4 ms for 1 (transfer 0x55560bd5af90)
* Expire in 4 ms for 1 (transfer 0x55560bd5af90)
* Expire in 5 ms for 1 (transfer 0x55560bd5af90)
*   Trying 172.217.16.138...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55560bd5af90)
* Connected to analytics.googleapis.com (172.217.16.138) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2633 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=upload.video.google.com
*  start date: Nov 10 14:39:49 2020 GMT
*  expire date: Feb  2 14:39:48 2021 GMT
*  subjectAltName: host "analytics.googleapis.com" matched cert's "*.googleapis.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55560bd5af90)
} [5 bytes data]
> GET /analytics/v3/metadata/ga/columns?alt=json HTTP/2
> Host: analytics.googleapis.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer ya29.a...
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 
< content-type: application/json; charset=UTF-8
< vary: X-Origin
< vary: Referer
< vary: Origin,Accept-Encoding
< date: Tue, 05 Jan 2021 14:11:13 GMT
< server: ESF
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< accept-ranges: none
<

and a request that was "stuck":

curl -4 --http2 --request GET 'https://analytics.googleapis.com/analytics/v3/metadata/ga/columns?alt=json' --header 'Authorization: Bearer ya29....' -v > /dev/null
Note: Unnecessary use of -X or --request, GET is already inferred.
* Expire in 0 ms for 6 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 0 ms for 1 (transfer 0x56189cda3f90)
* Expire in 0 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 0 ms for 1 (transfer 0x56189cda3f90)
* Expire in 0 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 1 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
* Expire in 2 ms for 1 (transfer 0x56189cda3f90)
*   Trying 172.217.22.42...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56189cda3f90)
* Connected to analytics.googleapis.com (172.217.22.42) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2633 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=upload.video.google.com
*  start date: Nov 10 14:39:49 2020 GMT
*  expire date: Feb  2 14:39:48 2021 GMT
*  subjectAltName: host "analytics.googleapis.com" matched cert's "*.googleapis.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x56189cda3f90)
} [5 bytes data]
> GET /analytics/v3/metadata/ga/columns?alt=json HTTP/2
> Host: analytics.googleapis.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer ya29....
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]

clmcavaney commented 3 years ago

Had the same issue. I can see from a nmap that TCP ports 80 and 443 are not open on the IPv6 addresses for www.googleapis.com but they are open for IPv4 addresses:

$ nmap -Pn -p80,443 -6 www.googleapis.com

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-04 17:57 AEST
Nmap scan report for www.googleapis.com (2404:6800:4015:802::200a)
Host is up.
Other addresses for www.googleapis.com (not scanned): 2404:6800:4015:801::200a 2404:6800:4015:800::200a 2404:6800:4015:803::200a 142.250.70.234 142.250.70.170 142.250.70.138 142.250.70.202
rDNS record for 2404:6800:4015:802::200a: mel05s01-in-x0a.1e100.net

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.07 seconds

$ nmap -4 www.googleapis.com

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-04 17:56 AEST
Nmap scan report for www.googleapis.com (142.250.70.202)
Host is up (0.034s latency).
Other addresses for www.googleapis.com (not scanned): 2404:6800:4015:802::200a 2404:6800:4015:803::200a 2404:6800:4015:800::200a 2404:6800:4015:801::200a 142.250.70.234 142.250.70.138 142.250.70.170
rDNS record for 142.250.70.202: mel05s01-in-f10.1e100.net
Not shown: 998 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 5.16 seconds

I used the quick hack from here in my code to force the underlying socket to connect over IPv4: https://stackoverflow.com/a/50044152

It looks like a Google firewall issue

glumia commented 3 years ago

Hi there! Just came to say that this issue (IPv6 connections that remain stuck until they timeout) still happens from time to time.

Last time I experienced it was on Thursday 7 November morning from ~9:00 to 10:15 (UTC+1) trying to connect to the address resolved by accounts.google.com.

huahouzwift commented 2 years ago

Looks like the issue still exists as a few developers from my organization experienced it now.

File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 920, in execute
    resp, content = _retry_request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/google_auth_httplib2.py", line 218, in request
    response, content = self.http.request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1708, in request
    (response, content) = self._request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1376, in _conn_request
    response = conn.getresponse()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 1345, in getresponse
    response.begin()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

sourdzl commented 2 years ago

have been experiencing this intermittently as well while using the gmail client, will try catching and retrying for now to see if that gets through it

valentin-panalyt commented 2 years ago

Issue still there for me.

rashivkp commented 2 years ago

@nulld mentioned a workaround in #563 where he had to move the socket.setdefaulttimeout() function before import googleapiclient. If it solves the problem, then credit goes to @nulld ! Either way, we can use this data point to better understand whether there are multiple issues going on. Please can someone who is experiencing the issue try the workaround and report back?

@parthea this workaround is working for me for downloading files using drive API.

bradyjensen1115 commented 1 year ago

Also getting this same issue... working code using the Sheets API just stopped. and now getting the socket timeout error every single time. I've tried oauth and API key and am seeing identical results.

I tried the workaround linked above and was unable to make progress.

Even in places where the code make it through (like authorizing the credentials), it take 2-5 minutes where before it was happening almost instantly (note: before meaning 'when the code worked' not 'before the workaround').

xEverth commented 6 months ago

Experiencing the same issue with the upload of 8.394 KB to a spreadsheet with BatchUpdate (20000 rows, 40 columns) even after preferring ipv4 in /etc/gai.conf The docs suggest to stay below 2MB for operations but state that otherwise there are no hard limits: https://developers.google.com/sheets/api/limits#quota Initiating the request from Germany The num_retries parameter is ignored in case of a socket timeout:

The httplib2 _conn_request method raises directly for a socket.timeout
The num_retries is used by the googleapiclient.discovery build inside _retrieve_discovery_doc but the returned discovery doc makes no mention of "retry" or "retries" in the response (at least for sheets v4). Do I have to pass num_retries to every execute() ?

Let me know if any other information can be useful for troubleshooting

googleapis / google-api-python-client

`socket.timeout: timed out` when using full IPv6 stack. #709

Environment details

Steps to reproduce