search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.46k stars 2.6k forks source link

Linux agent downloads artifact tool every time #14300

Open hbuckle opened 3 years ago

hbuckle commented 3 years ago

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: UniversalPackages@0

Environment

Issue Description

When running a build on a windows agent (windows-2019) with multiple UniversalPackages tasks the first task downloads the ArtifactTool and subsequent tasks use the cached version. Running the same build on the Linux agent the ArtifactTool is downloaded every time the UniversalPackage task runs.

Task logs

Windows task first instance:

==============================================================================
Task         : Universal packages
Description  : Download or publish Universal Packages
Version      : 0.180.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
==============================================================================
SYSTEMVSSCONNECTION exists true
Downloading: https://04pvsblobproduks120.vsblob.vsassets.io/artifacttool/artifacttool-win10-x64-Release_0.2.181.zip?sv=2019-07-07&sr=b&sig=Mtqm94EdV1tWQJ4xviMhotVk%2F6vgKe7PXppfPvrIAQw%3D&spr=https&se=2021-01-28T12%3A53%3A30Z&sp=r&P1=1611838110&P2=11&P3=2&P4=4Z2f%2fJSWdurddWMkrx85RWlqHWgyEPrv6TCVxVnSGmA%3d
Caching tool: ArtifactTool 0.2.181 x64
SYSTEMVSSCONNECTION exists true

Windows task second instance:

==============================================================================
Task         : Universal packages
Description  : Download or publish Universal Packages
Version      : 0.180.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
==============================================================================
SYSTEMVSSCONNECTION exists true
Found tool in cache: artifacttool 0.2.181 x64
SYSTEMVSSCONNECTION exists true

Linux task first instance:

==============================================================================
Task         : Universal packages
Description  : Download or publish Universal Packages
Version      : 0.180.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
==============================================================================
SYSTEMVSSCONNECTION exists true
Downloading: https://04pvsblobproduks120.vsblob.vsassets.io/artifacttool/artifacttool-linux-x64-Release_0.2.181.zip?sv=2019-07-07&sr=b&sig=0YjK8P3Njpxbg1D1pjMTrvOxO8M6Xdyq7EFLuNuVzkU%3D&spr=https&se=2021-01-28T13%3A01%3A28Z&sp=r&P1=1611838588&P2=11&P3=2&P4=MVefoGSr%2bwvaqCTLSNMyr%2b0p%2fz8Pe4CBwUprjLBiBvY%3d
Caching tool: ArtifactTool 0.2.181 x64
SYSTEMVSSCONNECTION exists true

Linux task second instance:

==============================================================================
Task         : Universal packages
Description  : Download or publish Universal Packages
Version      : 0.180.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
==============================================================================
SYSTEMVSSCONNECTION exists true
Downloading: https://04pvsblobproduks120.vsblob.vsassets.io/artifacttool/artifacttool-linux-x64-Release_0.2.181.zip?sv=2019-07-07&sr=b&sig=QwgFyNQ7DW9JiSeqvo28lSvlWVVFeUbV2JDSRFU7etI%3D&spr=https&se=2021-01-28T13%3A02%3A15Z&sp=r&P1=1611838635&P2=11&P3=2&P4=IVj%2b9t%2f5qJuvGZNsx%2bomuMQWURnlQF5VYDOecK0sWFA%3d
Caching tool: ArtifactTool 0.2.181 x64
SYSTEMVSSCONNECTION exists true
ikocev commented 3 years ago

Any light on this? It is quite annoying to download the same tool for multiple packages in single pipeline, the execution is totally random due to this sometimes even taking 5 minutes to download the ArtifactTool.

ShrinivasKR commented 3 years ago

I too have been encountering the issue. We are using self hosted Ubuntu-18.04 Linux agents. The task usually takes between 2-6 minutes. It also often hangs for much longer periods (Last instance was ~15 minutes).

==============================================================================
Task         : Universal packages
Description  : Download or publish Universal Packages
Version      : 0.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
==============================================================================
SYSTEMVSSCONNECTION exists true
Downloading: https://0t3vsblobprodcus362.vsblob.vsassets.io/artifacttool/artifacttool-linux-x64-Release_0.2.189.zip?sv=2019-07-07&sr=b&sig=zKsrxCSH0w92RidiftILYZv4YgCdCAU2LWGBMFMl4t4%3D&spr=https&se=2021-05-18T20%3A49%3A03Z&sp=r&P1=1621370643&P2=11&P3=2&P4=G5pC2%2fIcIwXoBqrglvY54BNzQczRp7kZshrFQNN4FZE%3d
Caching tool: ArtifactTool 0.2.189 x64

This is pretty much the log every time the task is run. It doesn't look like the package is being cached correctly. The download appears to be attempted every time, and the time it takes to normally complete randomly varies. The resolution of this issue would greatly improve pipeline build times.

wiktor-io commented 2 years ago

I noticed that the identifier used for caching the tool is inconsistent:

In UniversalPackagesV0 it's passed as lowercase artifacttool:

https://github.com/microsoft/azure-pipelines-tasks/blob/019f20abce0f61483356ea392916ccf6fd6672eb/Tasks/UniversalPackagesV0/universalmain.ts#L31-L35

Which is later used in ArtifactToolUtilities to read from the cache:

https://github.com/microsoft/azure-pipelines-tasks/blob/019f20abce0f61483356ea392916ccf6fd6672eb/Tasks/Common/packaging-common/universal/ArtifactToolUtilities.ts#L81

However, when writing to cache, it's hardcoded as capitalised ArtifactTool:

https://github.com/microsoft/azure-pipelines-tasks/blob/019f20abce0f61483356ea392916ccf6fd6672eb/Tasks/Common/packaging-common/universal/ArtifactToolUtilities.ts#L90

It occurs to me that an operating system using case sensitive paths may never get a cache hit.

ecrowe-aridhia commented 2 years ago

I can confirm that it's a case sensitivity issue as symlinking /agent/_work/_tool/ArtifactTool and /agent/_work/_tool/artifacttool on our build agents allowed it to use the cached copy for me.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

hbuckle commented 2 years ago

Not stale - still needs fixing

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

hbuckle commented 1 year ago

Still not stale - still needs fixing

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

hbuckle commented 1 year ago

Not stale, someone should review the linked PR

nickidw commented 10 months ago

This is still an issue, 2 and a half years later. Can someone please look into it?

nickidw commented 10 months ago

Workaround: go to azagent/_work/_tool/ and run ln -s ArtifactTool artifacttool

mustafacagataytulun commented 4 months ago

This problem still exists.

I can also confirm that the workaround mentioned by @ecrowe-aridhia and @nickidw works on our self-hosted Ubuntu-22.04 agents. We saved a minimum of 10-11 seconds on each of our pipelines (uploading artifact) and releases (downloading artifact).