azcopy pending for a long time and no progress updated

[sam-wang-1991]

We used azcopy to download a 396GB big file from Azure blog storage yesterday, with 50 mbps limit.  However,No progress was updating 3 hours after azcopy was started.It looked like it was hung and pending forever. And the progress was always 0.0%.  but it was working fine last month when the file size was a little smaller 386GB using the same command.

The comomand we used is as follow.

azcopy copy https://storageaccount.blob.core.chinacloudapi.cn/container/bigfile?sp=sastoken /TargetDir/bigfile --cap-mbps 50  From the log file we could find the same contents kept generating.

2023/02/07 07:10:31 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:33 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:33 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:35 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:35 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:37 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:37 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:39 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:39 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:41 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:41 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:43 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:43 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:45 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:45 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:47 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:47 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:49 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:49 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:51 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300 2023/02/07 07:10:51 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2023/02/07 07:10:53 PERF: primary performance constraint is Unknown. States: C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs: 300

If we remove the --cap-mbps 50 option, azcopy works well. we could see progress updated in a short time. If we increase the value to 60mpbs , it's working also.  Could you explain the reason beyhind that? In addtion,  May I know what these letters stand for?  C:  0, R:  1, W: 468, F:  0, H:  0, B: 300, S:  0, P:  0, Q:  0, D:  0, E:  0, T: 769, GRs

[adreed-msft]

Low throughput caps can cause AzCopy to hang as it cannot allocate enough bandwidth to send a single block. Ensure your block size is smaller than your throughput cap.

[oasaleh]

Well, at least for me, I was downloading to a file share that its disk was limiting. I switched to a better disk and it worked fine.

[ZEB1CLJ]

I have this issue too with smaller files. I run azcopy as the final step of an Azure DevOps job and although the folder size is about only 600 MB, azcopy never finishes the job. It sometimes fails after 50% percent of the job, sometimes after 90% of the job. It behaves the same with larger folders (cca. 14 GB or 17 GB).

[evanderkarr]

I have also come across this issue when downloading large files ( >100GB) from US East to users in other regions. It seemed to only happen when using Azure Storage Explorer, but it had the exact same behavior in the logs. No complaints or notification of a failure, just a pending job sitting there forever until someone notices the percent completion is not progressing. It was reproduced by three different users many times in a row (there is no bandwidth capping when issue arises, so the block-size bandwidth allocation relationship should not be relevant).

(Same issue I posted here in the help forum https://learn.microsoft.com/en-us/answers/questions/1625028/azure-storage-explorer-hanging-after-attempting-a)

[Karishma-Tiwari-MSFT]

@adreed-msft There have been a few customers facing this issue. Has there been any update since Feb 2023 (your last comment), that can help customers manage this issue better? Looking for your insights on this. Thanks.