search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.97k stars 2.95k forks source link

Azure Universal Artifacts download of large package fails #16956

Open srothery opened 3 years ago

srothery commented 3 years ago

Describe the bug Downloading a Universal Package of > 22GB fails when using az cli and self hosted agent.

To Reproduce I uploaded a large universal package of around 22GB to a cloud hosted package feed. This took an appropriate amount of time based on my ISP upload speed. Using the following command then fails: az artifacts universal download --organization https://dev.azure.com/XXX --project="Test" --scope project --feed TestFeed --name ansible-media3 --version 2.3.2 --path fred It does seem to spend a good 30 minutes downloading what must be the majority of the package before failing with a 403 authorisation error.

Expected behavior I would expect a universal package of any size up to the reported max of 1TB (here) to download given sufficient time and space on the host.

Environment summary Using Windows 10 powershell and: azure-cli 2.18.0 core 2.18.0 telemetry 1.0.6 Extensions: azure-devops 0.18.0

Additional context There is a lot of information in an azure devops ticket here. A quick summary of what was done. We switched from using az login to using a PAT which didn't help. We then moved to self hosted agents which also gave the same error. We tried different accounts within the same organisation and this also didn't help. In the same environment repeated downloads of 1GB and 4GB universal packages are fine. My goal is to actually use universal packages of up to 100GB but this 22GB package was my third test of the capability. We can't adopt Azure Universal Packages if the download fails.

I've attached downloadexception.txt which was the initial error, downloadwithpat.txt using a PAT and then downloadlogwithdebug.txt which has the debug flag on.

Have people downloaded large universal packages (>22GB) with the CLI? I'm wondering if there is maybe some timeout option somewhere which means I get 30ish minutes of download before it fails?

Any help would be much appreciated. If this is the wrong place to raise this issue please let me know where to log it as the Azure Devops support after some time suggested here.

yonzhan commented 3 years ago

artifacts

srothery commented 3 years ago

Hi - have I created the issue correctly? I'm needing help on this or I can't move forward adopting Azure Arifacts to store my binary packages. I've already spent many weeks following steps on the azure devops community portal ticketing system to only be told at the end I should have raised it here...

srothery commented 3 years ago

Still needing help on this one if anyone has met this issue before

jamesnswithers commented 2 years ago

@srothery followed your (frustrating) ticket here! Did you have any luck in a fix or workaround or did it end up ignored?

srothery commented 2 years ago

Hi – yeah its very frustrating. What I believe to be the case is there is a hidden (fixed) timeout after which if the download/upload hasn’t complete it will fail. I’ve recently received a internet upgrade so most/all of the artifacts now can be downloaded within this 30 minute ish timeout. My advise would be to split up very large artifacts (anyway) as this then bypasses the timeout but also makes them more manageable. Good luck ! Simon

jamesnswithers commented 2 years ago

Fair. Seems the AZ CLI downloads some standalone tool which does the real downloading, so not sure this is even the right place to raise the issue :( !

srothery commented 2 years ago

I think this is the right place but the even more frustrating thing is that here in github there seems to be no MS support running things (this ticket is a good example). On the Azure Devops support there is much faster response time though of course the likely outcome of that is a closed ticket with no resolution