Azure CLI downloads are taking a long period of time due to time outs

[bbi-adamvengroff]

Related command az pipelines runs artifact download --artifact-name {name} --path c:/temp --run-id {id} --org "{org}" --project {proj} --debug

Describe the bug We are downloading the artifact fine but sometimes it stalls out and takes ~18 minutes instead of the expected ~3 minutes Looking at the output, I see

cli.azext_devops.dev.common.artifacttool: Downloaded 3,226.0 MB out of 3,302.6 MB (98%).
cli.azext_devops.dev.common.artifacttool: ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodscussu5.vsblob.visualstudio.com/A0bd78a1e-8bc2-4825-baee-2e3c7488e4c7/_apis/dedup/urls attempt 1/6 failed with TimeoutException: 'The HTTP request timed out after 00:15:00.'
cli.azext_devops.dev.common.artifacttool: Downloaded 3,226.0 MB out of 3,302.6 MB (98%).

The download then continues and successfully completes So it looks like it's a case of something timing out and then it waiting 15 minutes instead of something more reasonable (in the seconds)

To Reproduce Run above command. It happens with 100% repro for some team members, 0% for others and some get it occasionally. Maybe it's exasperated by network jitter?

Expected behavior Download should complete in a timely manner and timeouts should cause a small delay

Environment summary Install method: Web browsers CLI Version:

{
  "azure-cli": "2.32.0",
  "azure-cli-core": "2.32.0",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {
    "azure-devops": "0.22.0"
  }
}

OS Version: Windows10 (Version 10.0.19041 Build 19041) Shell Type: cmd.exe

[yonzhan]

route to CXP team

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @v-anvashist, @V-hmusukula.

Issue Details

---

**Related command** `az pipelines runs artifact download --artifact-name {name} --path c:/temp --run-id {id} --org "{org}" --project {proj} --debug` **Describe the bug** We are downloading the artifact fine but sometimes it stalls out and takes ~18 minutes instead of the expected ~3 minutes Looking at the output, I see ``` cli.azext_devops.dev.common.artifacttool: Downloaded 3,226.0 MB out of 3,302.6 MB (98%). cli.azext_devops.dev.common.artifacttool: ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodscussu5.vsblob.visualstudio.com/A0bd78a1e-8bc2-4825-baee-2e3c7488e4c7/_apis/dedup/urls attempt 1/6 failed with TimeoutException: 'The HTTP request timed out after 00:15:00.' cli.azext_devops.dev.common.artifacttool: Downloaded 3,226.0 MB out of 3,302.6 MB (98%). ``` The download then continues and successfully completes So it looks like it's a case of something timing out and then it waiting 15 minutes instead of something more reasonable (in the seconds) **To Reproduce** Run above command. It happens with 100% repro for some team members, 0% for others and some get it occasionally. Maybe it's exasperated by network jitter? **Expected behavior** Download should complete in a timely manner and timeouts should cause a small delay **Environment summary** Install method: Web browsers CLI Version: ``` { "azure-cli": "2.32.0", "azure-cli-core": "2.32.0", "azure-cli-telemetry": "1.0.6", "extensions": { "azure-devops": "0.22.0" } } ``` OS Version: Windows10 (Version 10.0.19041 Build 19041) Shell Type: cmd.exe

Author:	bbi-adamvengroff
Assignees:	-
Labels:	`Service Attention`, `customer-reported`, `DevOps`, `Pipelines`, `Auto-Assign`
Milestone:	Backlog

[navba-MSFT]

@bbi-adamvengroff Adding Service Team for the Devops issue

@v-anvashist, @V-hmusukula Could you please look into this issue and provide an update ? Awaiting your reply.

[v-soujanya]

Hi @bbi-adamvengroff, we tried to repro the issue on our end, but we couldn't successful. We are able to download artifacts successfully. To look into further could you please raise a support ticket on the same and please share the artifacts details which can help us to investigate better. Thanks!

[bbi-adamvengroff]

Hi @bbi-adamvengroff, we tried to repro the issue on our end, but we couldn't successful. We are able to download artifacts successfully. To look into further could you please raise a support ticket on the same and please share the artifacts details which can help us to investigate better. Thanks!

Could you elaborate on what you mean by sharing the artifact details? as in the specific artifacts we are trying to download?

[v-soujanya]

@bbi-adamvengroff, we have tried some artifacts and got downloaded in couple of seconds and, the size of artifacts is not huge which we downloaded, we are trying to find some publicly available artifacts of size equivalent to 3GB which is nearly equivalent to your's artifacts. The meaning of sharing artifacts details is the package which you are trying to download if you will provide access to us.

[bbi-adamvengroff]

@bbi-adamvengroff, we have tried some artifacts and got downloaded in couple of seconds and, the size of artifacts is not huge which we downloaded, we are trying to find some publicly available artifacts of size equivalent to 3GB which is nearly equivalent to your's artifacts. The meaning of sharing artifacts details is the package which you are trying to download if you will provide access to us.

Ah yeah I definitely can't share the artifact we are trying to download. I'm guessing you guys can't make a pipeline that just uploads a 3gbs of dummy files? That said, it's hard to repro the issue - are you unable to look at ArtifactHttpRetryMessageHandler.SendAsync's code to check for a 15 minute timeout?

[v-soujanya]

Hi @bbi-adamvengroff, we are not able to find this ArtifactHttpRetryMessageHandler.SendAsync in our code because we are using the artifact tool to download the artifacts. By seeing this https://vsblobprodscussu5.vsblob.visualstudio.com/A0bd78a1e-8bc2-4825-baee-2e3c7488e4c7/_apis/dedup/urls attempt 1/6 failed with Timeout Exception We identified the issue is causing at server end from CUS region, which is causing time out issue, but still, we are trying to figure out the issue by checking some docs as we are not able to identify this in our code.

[bbi-adamvengroff]

Hi @bbi-adamvengroff, we are not able to find this ArtifactHttpRetryMessageHandler.SendAsync in our code because we are using the artifact tool to download the artifacts. By seeing this https://vsblobprodscussu5.vsblob.visualstudio.com/A0bd78a1e-8bc2-4825-baee-2e3c7488e4c7/_apis/dedup/urls attempt 1/6 failed with Timeout Exception We identified the issue is causing at server end from CUS region, which is causing time out issue, but still, we are trying to figure out the issue by checking some docs as we are not able to identify this in our code.

Huh, interesting. Thanks for the update! Please let me know if I can be of assistance