generator hanging sometimes using list_blobs() or list_blob_names()

Master-Matt commented 5 years ago

Which service(blob, file, queue) does this issue concern?

BlockBlobService

Which version of the SDK was used? Please provide the output of `pip freeze`.

asn1crypto==0.24.0 azure-common==1.1.19 azure-storage-blob==1.5.0 azure-storage-common==1.4.0 certifi==2019.3.9 cffi==1.12.2 chardet==3.0.4 coverage==4.0.3 cryptography==2.6.1 entrypoints==0.3 fastavro==0.21.21 flake8==3.7.7 idna==2.8 influx-client==1.8.0 influxdb==5.2.2 mccabe==0.6.1 mock==2.0.0 nose==1.3.7 numpy==1.16.2 pandas==0.24.2 pandavro==1.5.0 pbr==5.1.3 pycodestyle==2.5.0 pycparser==2.19 pyflakes==2.1.1 python-dateutil==2.8.0 pytool==3.15.0 pytz==2019.1 requests==2.21.0 simplejson==3.16.0 six==1.12.0 urllib3==1.24.2

What problem was encountered?

I'm using this service in a docker container, and it will occasionally hang for up to 30 minutes. Not sure what to do because it happens as the blobs are iterated through.

I had the same issue when creating the generator, but it can be mitigated by using the timeout option. Iv'e attached the trace.

Traceback (most recent call last):
  File "data_service.py", line 200, in <module>
    main()
  File "data_service.py", line 195, in main
    if ds.get_data():
  File "data_service.py", line 38, in get_data
    for blob in generator:
  File "/usr/local/lib/python3.5/site-packages/azure/storage/common/models.py", line 99, in __iter__
    resources = self._list_method(*self._list_args, **self._list_kwargs)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/blob/baseblobservice.py", line 1375, in _list_blobs
    return self._perform_request(request, _converter, operation_context=_context)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/common/storageclient.py", line 321, in _perform_request
    response = self._httpclient.perform_request(request)
  File "/usr/local/lib/python3.5/site-packages/azure/storage/common/_http/httpclient.py", line 92, in perform_request
    proxies=self.proxies)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.5/http/client.py", line 1198, in getresponse
    response.begin()
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.5/socket.py", line 576, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.5/ssl.py", line 937, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.5/ssl.py", line 799, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/lib/python3.5/ssl.py", line 583, in read
    v = self._sslobj.read(len, buffer)

Here is a shot of VM usage to show that it looks like its just idling after the hangs occur. Here, it occurs at 12:22 and 12:42 Screenshot from 2019-06-24 13-12-45

Have you found a mitigation/solution?

No

zezha-msft commented 5 years ago

Hi @Master-Matt, thanks for reaching out!

To confirm, is your VM running in Azure? If yes, is it in the same region as the Storage account you are listing?

The listing generator is making calls to the service lazily to fetch batches of blobs. I suspect that sometimes these calls take longer than expected, and get retried. Perhaps you could turn on logging and see if this is the case.

A potential workaround is to set the socket_timeout on the service object to a shorter time:

:param int socket_timeout: If specified, this will override the default socket timeout. The timeout specified is in seconds. See DEFAULT_SOCKET_TIMEOUT in _constants.py for the default value.

Master-Matt commented 5 years ago

@zezha-msft Yes, everything is in Azure. I am not in the same region as the storage account. Storage is EU and I'm in US. I will turn on debugging and grab some output. It's a lot of spam since there are 40k+ blobs to go through every time I call this. I will also try out setting the timeout. Thanks!!

Master-Matt commented 5 years ago

So the issue was that, since I'm using python 3.5, the timeout is a tuple of (20,2000) where the 2000 is the read timeout (about 30 minutes) I shortened it down quite a bit. Thanks for the help!

zezha-msft commented 5 years ago

@Master-Matt sounds good! Please don't hesitate if you have any other question or issue. 😄

Azure / azure-storage-python