Blob upload fails on (not very) large EXE file

sicklittlemonkey commented 3 years ago

This is autogenerated. Please review and update as needed.

Describe the bug

A 56 MiB EXE file fails to upload. This was working for me from months ago to a week or two ago, but now I can only upload files up to 38 MiB or so in size. The main problem is that the error message doesn't tell me anything useful. Using azcopy 10.6.0 the larger file can be successfully copied to the destination. Does azcopy just have better error handling than az cli?

Command Name az storage blob upload

Errors: The unhelpful one is:

HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-04-08&se=2021-07-30T06%3A53%3A11Z&sr=c&sp=rwl&sig=xpnoC8%[...]%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)')))

Sometimes I get the slightly more helpful timeout one:

ERROR: Client-Request-ID=dd3a6552-da5e-11eb-8c17-04d4c4a9454f Retry policy did not allow for a retry: , HTTP status code=Unknown, Exception=('Connection aborted.', timeout('The write operation timed out')).
ERROR: ('Connection aborted.', timeout('The write operation timed out'))

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

I don't know - just upload a big file. Nothing has changed in my environment. Changing the SAS token didn't help.
az storage blob upload --debug -f {} -c {} -n {}

Expected Behavior

For files smaller than 40-50 MiB it just shows the progress bar and uploads:

Finished[#############################################################]  100.0000%

But a 56 MiB file which worked fine before, now gets an unhelpful double error like this:

Client-Request-ID=cde0e303-d965-11eb-adb6-04d4c4a9454f Retry policy did not allow for a retry: , HTTP status code=Unknown, Exception=HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-02-10&ss=bfqt&srt=sco&sp=rwdlacupx&se=2022-04-09T13%3A43%3A07Z&st=2021-04-09T05%3A43%3A07Z&spr=https&sig=[...]%2BnV4%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)'))).
HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-02-10&ss=bfqt&srt=sco&sp=rwdlacupx&se=2022-04-09T13%3A43%3A07Z&st=2021-04-09T05%3A43%3A07Z&spr=https&sig=[...]%2BnV4%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)')))

Environment Summary

Windows-10-10.0.19041-SP0
Python 3.8.9
Installer: MSI

azure-cli 2.25.0

Extensions:
azure-devops 0.18.0

Additional Context

yonzhan commented 3 years ago

storage

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details

### **This is autogenerated. Please review and update as needed.** ## Describe the bug A 56 MiB EXE file fails to upload. This was working for me from months ago to a week or two ago, but now I can only upload files up to 38 MiB or so in size. The main problem is that the error message doesn't tell me anything useful. Using azcopy 10.6.0 the larger file can be successfully copied to the destination. Does azcopy just have better error handling than az cli? **Command Name** `az storage blob upload` **Errors:** The unhelpful one is: ``` HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-04-08&se=2021-07-30T06%3A53%3A11Z&sr=c&sp=rwl&sig=xpnoC8%[...]%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)'))) ``` Sometimes I get the slightly more helpful timeout one: ``` ERROR: Client-Request-ID=dd3a6552-da5e-11eb-8c17-04d4c4a9454f Retry policy did not allow for a retry: , HTTP status code=Unknown, Exception=('Connection aborted.', timeout('The write operation timed out')). ERROR: ('Connection aborted.', timeout('The write operation timed out')) ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - I don't know - just upload a big file. Nothing has changed in my environment. Changing the SAS token didn't help. - `az storage blob upload --debug -f {} -c {} -n {}` ## Expected Behavior For files smaller than 40-50 MiB it just shows the progress bar and uploads: ``` Finished[#############################################################] 100.0000% ``` But a 56 MiB file which worked fine before, now gets an unhelpful double error like this: ``` Client-Request-ID=cde0e303-d965-11eb-adb6-04d4c4a9454f Retry policy did not allow for a retry: , HTTP status code=Unknown, Exception=HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-02-10&ss=bfqt&srt=sco&sp=rwdlacupx&se=2022-04-09T13%3A43%3A07Z&st=2021-04-09T05%3A43%3A07Z&spr=https&sig=[...]%2BnV4%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)'))). HTTPSConnectionPool(host='storageshare.blob.core.windows.net', port=443): Max retries exceeded with url: /path1/path2/filename.exe?sv=2020-02-10&ss=bfqt&srt=sco&sp=rwdlacupx&se=2022-04-09T13%3A43%3A07Z&st=2021-04-09T05%3A43%3A07Z&spr=https&sig=[...]%2BnV4%3D (Caused by SSLError(SSLWantWriteError(3, 'The operation did not complete (write) (_ssl.c:2479)'))) ``` ## Environment Summary ``` Windows-10-10.0.19041-SP0 Python 3.8.9 Installer: MSI azure-cli 2.25.0 Extensions: azure-devops 0.18.0 ``` ## Additional Context

Author:	sicklittlemonkey
Assignees:	Juliehzl
Labels:	`Service Attention`, `Storage`
Milestone:	Jul 2021 (2021-08-03)

jfro commented 2 years ago

I'm seeing similar error though not tied to file size or anything obvious. I uploaded some files fine, but another file would struggle to be uploaded with errors described in this issue (timeout etc) I'd upload 80M or so files fine, then go to upload this 47M file and it'd consistently fail, eventually with enough trying I finally get it uploaded.

Azure / azure-cli