microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.42k stars 2.58k forks source link

[BUG]: Azure File transform task. EPERM: operation not permitted, stat 'C:\agent\_work\r1\a\build_api\Jobs\App_Data\jobs\continuous\protocols\bin\2d35ec66-b8b4-4f9b-8a50-753889deb977.tmp' #19933

Open yuriykuts opened 1 month ago

yuriykuts commented 1 month ago

New issue checklist

Task name

File transform

Task version

1.235.1

Issue Description

We are using the File Transform task to substitute variables in a JSON file within the release pipeline, and we are encountering an issue with the error:

[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'

This error is reproducible on different agents.

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2019 Standart

Relevant log output

2024-05-30T13:40:37.9775830Z ##[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9785834Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9788081Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9987040Z ##[section]Finishing: Jobs configuration transform

Full task logs with system.debug enabled

2024-05-30T13:40:35.2226188Z ##[debug]Evaluating condition for step: 'Jobs configuration transform' 2024-05-30T13:40:35.2228803Z ##[debug]Evaluating: succeeded() 2024-05-30T13:40:35.2229335Z ##[debug]Evaluating succeeded: 2024-05-30T13:40:35.2230240Z ##[debug]=> True 2024-05-30T13:40:35.2231883Z ##[debug]Result: True 2024-05-30T13:40:35.2232957Z ##[section]Starting: Jobs configuration transform 2024-05-30T13:40:35.3127120Z ============================================================================== 2024-05-30T13:40:35.3127540Z Task : File transform 2024-05-30T13:40:35.3127712Z Description : Replace tokens with variable values in XML or JSON configuration files 2024-05-30T13:40:35.3128012Z Version : 1.235.1 2024-05-30T13:40:35.3128303Z Author : Microsoft Corporation 2024-05-30T13:40:35.3128491Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform 2024-05-30T13:40:35.3128783Z ============================================================================== 2024-05-30T13:40:37.9664033Z ##[debug] C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin (directory) 2024-05-30T13:40:37.9727460Z ##[debug]task result: Failed 2024-05-30T13:40:37.9775830Z ##[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp' 2024-05-30T13:40:37.9785834Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp' 2024-05-30T13:40:37.9788081Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp' 2024-05-30T13:40:37.9987040Z ##[section]Finishing: Jobs configuration transform

Repro steps

No response

paulcurren commented 5 days ago

The problem appears to arise in the download-pipeline-artifact task.

The artefacts are downloaded as chunks - the '.tmp' files. It seems that the download task exits before all the '.tmp' files are uncompressed into real files. The uncompressing carries on in the background after the download task has exited.

Adding a PowerShell script to add a 60 second delay after download will allow the next task to complete.

This is not a 'workaround'. The issue with the download task needs to be fixed.

paulcurren commented 5 days ago

Had to add a PowerShell script in a new pipeline stage to make sure the downloads are complete before allowing further deployment pipeline stages to complete:

while ($true)
{
 $tmpFileCount = (Get-ChildItem -Path MyWebApp -Filter *.tmp -Recurse).Count

 if ($tmpFileCount -eq 0)
 {
   break;
 }

 Write-Host "Waiting for $tmpFileCount .tmp files..."
 Start-Sleep -Seconds 5
}

Result:

2024-06-27T10:42:41.6678367Z Waiting for 4 .tmp files...
2024-06-27T10:42:46.6956047Z Waiting for 3 .tmp files...
2024-06-27T10:42:51.7231092Z Waiting for 3 .tmp files...
2024-06-27T10:42:56.7486519Z Waiting for 3 .tmp files...
2024-06-27T10:43:01.7796169Z Waiting for 1 .tmp files...
2024-06-27T10:43:06.7994872Z Waiting for 1 .tmp files...
2024-06-27T10:43:11.8147468Z Waiting for 1 .tmp files...
2024-06-27T10:43:16.8391393Z Waiting for 1 .tmp files...
2024-06-27T10:43:21.8675132Z Waiting for 1 .tmp files...
2024-06-27T10:43:26.8840815Z Waiting for 1 .tmp files...
2024-06-27T10:43:31.9054132Z Waiting for 1 .tmp files...
2024-06-27T10:43:36.9249268Z Waiting for 1 .tmp files...
2024-06-27T10:43:41.9504679Z Waiting for 1 .tmp files...

This is a kludge, not a workaround.