Blob upload gets stuck after two consecutive upload attempts

[ppessi]

• Package Name: azure-storage-blob
• Package Version: 12.3.2
• Operating System: Windows 10
• Python Version: 3.8.1

Describe the bug My program attempts to re-upload a blob with a different name if the first upload attempt fails with a ResourceExistsError. It seems that the second upload never succeeds, and the program responds very slowly to a KeyboardInterrupt. I think the longest time I have waited for the upload to complete is several minutes. The blob is not big, the upload should only take a few seconds at most. This might also be a problem at Azure's end, of course, but two upload attempts withing a span of a couple seconds shouldn't be enough to cause any performance issues...

The code that attempts to upload the blob:

with open(blob, 'rb') as data:
    n = 0
    while True:
        blob_name = "%s_%d%s" % (parts[0], n, parts[1])
        print("Attempting to upload to %s/%s" % (container, blob_name))
        with blob_service_client.get_blob_client(
                container=container, blob=blob_name) as blob_client:
            try:
                blob_client.upload_blob(data, timeout=5)
                print("Upload successful")
                return
            except ResourceExistsError:
                print("Blob already exists")
                n += 1
            except:
                print("Upload to blob failed:\n", format_exc())
                return

The output:

Attempting to upload to output/running-example_0.bpmn
Blob client created
Attempting to upload to output/running-example_1.bpmn
Blob client created

After that it gets stuck. If I use ctrl+c, it shows the following traceback after some time, a minute or two, perhaps.

Upload to blob failed:
 Traceback (most recent call last):
  File ".\__main__.py", line 103, in upload_blob
    blob_client.upload_blob(data, timeout=5)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_blob_client.py", line 496, in upload_blob
    return upload_block_blob(**options)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_upload_helpers.py", line 94, in upload_block_blob
    return client.upload(
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_generated\operations\_block_blob_operations.py", line 207, in upload
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 157, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 489, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 290, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 103, in send
    self._sender.send(request.http_request, **request.context.options),
  File "~\ProjectFolder\.venv\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 312, in send
    return self._transport.send(request, **kwargs)
  File "~\ProjectFolder\.venv\lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 259, in send
    response = self.session.request(  # type: ignore
  File "~\ProjectFolder\.venv\lib\site-packages\requests\sessions.py", line 530, in request
  File "~\ProjectFolder\.venv\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "~\ProjectFolder\.venv\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "~\ProjectFolder\.venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "~\ProjectFolder\.venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "~\ProjectFolder\.venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "~\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1322, in getresponse
    response.begin()
  File "~\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "~\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "~\AppData\Local\Programs\Python\Python38\lib\socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "~\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "~\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
KeyboardInterrupt

To Reproduce Steps to reproduce the behavior:

1. Attempt to upload to a blob that already exists
2. Try again with a different name

Expected behavior It should upload the blob on the second try (if the name is available).

Additional context This is more for handling outlier situations, the frontend should be taking care that there won't be blobs with the same names.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[xiafu-msft]

Hi @ppessi

Thanks for reaching out.

The SDK has some default retry, https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/storage/azure-storage-blob/README.md#retry-policy-configuration SDK will wait some time before retry. You can specify retry_total=0 to stop retry since your code has retry.

Let me know if that doesn't help! Thanks

[ppessi]

@xiafu-msft I did that and waited a couple minutes (not sure if I ever waited that long before) and got this error.

 Traceback (most recent call last):
  File ".\__main__.py", line 103, in upload_blob
    blob_client.upload_blob(data, retry_total=0)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_blob_client.py", line 496, in upload_blob
    return upload_block_blob(**options)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_upload_helpers.py", line 94, in upload_block_blob
    return client.upload(
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_generated\operations\_block_blob_operations.py", line 207, in upload
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  [Previous line repeated 5 more times]
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 157, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 515, in send
    raise err
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 489, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 290, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 103, in send
    self._sender.send(request.http_request, **request.context.options),
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 312, in send
    return self._transport.send(request, **kwargs)
  File "C:\Users\PinjaJäkkö\Documents\koodeja\PM4Py_Docker\.venv\lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 284, in send
    raise error
azure.core.exceptions.ServiceResponseError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

[xiafu-msft]

Hi @ppessi

It looks like a read timeout. Were you able to upload any blob?

[ppessi]

Yes, it works fine except if I try to do it two times in a row.

[xiafu-msft]

Hi @ppessi

Thanks for the info, while I still have two questions and I need some info to reproduce it successfully.

If you’ve uploaded successfully, then “ Upload successful” should be printed out right? I’m just assuming the console has all output info.

Does “Do it twice in a row” mean uploading to the same blob twice or to two different blobs?

[ppessi]

Ah, sorry for being unclear, I meant that when there is no blob of the same name already it does succeed. This problem only occurs when a blob with the same name already exists and I try to create a new one with a different name. So first I attempt to upload to blob A, then to blob B, and it fails to upload to B.

[xiafu-msft]

Hi @ppessi

Thanks for clarifying this! I will try to reproduce it today and see if I have any clue!

[xiafu-msft]

Hi @ppessi

I wasn't able to reproduce the hanging problem by uploading to the same block blob twice.

If it's an read timeout, can you increase the read_timeout value? If you are using python2 let's increase it to 40, if you are using python3 then it shouldn't be read timeout. Can you add HttpResponseError (as the code below) to see if can help catch any other error? Also can you confirm the behavior is what you want? Thanks a lot for helping locate the problem!

Based on your code, let's assume parts[0] == "a" and parts[1] == "b" loop 0: blob_name = a_0b loop 1: blob_name = a_0b throws resource exists exception n = 1 loop 2: blob_name = a_1b loop 3: blob_name = a_1b throws resource exists exception n = 2 loop 4: blob_name = a_2b loop 4: blob name = a_2b throws resource exists exception n = 3 ............ so on

with open(blob, 'rb') as data:
    n = 0
    while True:
        blob_name = "%s_%d%s" % (parts[0], n, parts[1])
        print("Attempting to upload to %s/%s" % (container, blob_name))
        with blob_service_client.get_blob_client(
                container=container, blob=blob_name) as blob_client:
            try:
                blob_client.upload_blob(data, timeout=5)
                print("Upload successful")
                return
            except ResourceExistsError:
                print("Blob already exists")
                n += 1
            except:
                print("Upload to blob failed:\n", format_exc())
                return
           except HttpResponseError as e:
                print("This is an unexpected error")
                print(e)

[ppessi]

Hmmm it started working now even though I didn't change anything significant?? I'm using a different wi-fi than before, could that affect it...? Nothing should've changed on the storage account's side, either.

And yeah the behavior I want is that it continues trying until it finds a blob that doesn't exist and then uploads that. I don't really know if this is the best way to do that but it's for handling edge cases, anyway.

[xiafu-msft]

Hi @ppessi

It looks like you don't encounter the problem anymore, is it good for you if we close the issue now?

[ppessi]

Well it's still a mystery why it didn't work at first but not sure there's anything that can be debugged... but it does work now.