search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.5k stars 2.61k forks source link

[BUG]: Exit 57005 StackOverflowException using AzureFileCopy@6 #20221

Open directhex opened 3 months ago

directhex commented 3 months ago

New issue checklist

Task name

AzureFileCopy

Task version

6.243.0

Issue Description

We've been experiencing intermittent issues with StackOverflowException in this task. It seems very likely to be related to https://github.com/microsoft/azure-pipelines-tasks/issues/20156

We've experienced this failure on 6.243.0, 6.242.10, and 6.242.6 (I can't find it happening on older versions, but our job is only daily so that's not a guarantee)

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2022

Relevant log output

2024-07-29T12:05:44.3955659Z ##[section]Starting: Upload index to Azure Stroage
2024-07-29T12:05:44.3968038Z ==============================================================================
2024-07-29T12:05:44.3968174Z Task         : Azure file copy
2024-07-29T12:05:44.3968233Z Description  : Copy files to Azure Blob Storage or virtual machines
2024-07-29T12:05:44.3968327Z Version      : 6.243.0
2024-07-29T12:05:44.3968383Z Author       : Microsoft Corporation
2024-07-29T12:05:44.3968488Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2024-07-29T12:05:44.3968585Z ==============================================================================
2024-07-29T12:05:45.8990841Z Added TLS 1.2 in session.
2024-07-29T12:05:46.2175009Z ##[command]Import-Module -Name C:\Modules\az_11.3.1\Az.Accounts\3.0.2\Az.Accounts.psd1 -Global
2024-07-29T12:05:49.7890152Z ##[command]Update-AzConfig -DisplayBreakingChangeWarning False -AppliesTo Az
2024-07-29T12:05:49.9407665Z 
2024-07-29T12:05:50.0271404Z ##[command]Enable-AzureRmAlias -Scope Process
2024-07-29T12:05:51.2715650Z 
2024-07-29T12:05:51.2716199Z Process is terminated due to StackOverflowException.
2024-07-29T12:06:11.1659556Z ##[error]Exit code 57005 returned from process: file name 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', arguments '-NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". ([scriptblock]::Create('if ([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) { [Console]::InputEncoding = New-Object Text.UTF8Encoding $false } if (!$PSHOME) { $null = Get-Item -LiteralPath ''variable:PSHOME'' } else { Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Management\Microsoft.PowerShell.Management.psd1'')) ; Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psd1'')) }')) 2>&1 | ForEach-Object { Write-Verbose $_.Exception.Message -Verbose } ; Import-Module -Name 'D:\a\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.243.0\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1' -ArgumentList @{ NonInteractive = $true } -ErrorAction Stop ; $VerbosePreference = 'SilentlyContinue' ; $DebugPreference = 'SilentlyContinue' ; Invoke-VstsTaskScript -ScriptBlock ([scriptblock]::Create('. ''D:\a\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.243.0\AzureFileCopy.ps1'''))"'.
2024-07-29T12:06:11.1674268Z ##[section]Finishing: Upload index to Azure Stroage

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION]

Repro steps

- task: AzureFileCopy@6
          displayName: Upload index to Azure Stroage
          inputs:
            azureSubscription: SourceDotNet-Deployment-ARM
            SourcePath: "bin/index/index/*"
            Destination: AzureBlob
            storage: $(storageAccount)
            ContainerName: $(NEW_CONTAINER_NAME)
EshitaD commented 2 months ago

Hi @directhex, is this issue still persists?

directhex commented 2 months ago

Actually, I haven't seen this failure personally in August. Let me check in with a Teams chat w/ other people suffering from related issues.

Schniefel commented 1 month ago

The issue still exists

v-schhabra commented 1 month ago

Hi @directhex

  1. What kind of ARM service endpoint are you using in AzureFileCopy tasks?
  2. Where is the build agent running?
  3. How does the whole pipeline definition looks like?
  4. Share the complete debug logs?

Please share these details in order to investigate further.

directhex commented 1 month ago

I haven't had the issues since July 29th, on a daily job. I can't guarantee I haven't just been very lucky, but from my perspective, I'm no longer experiencing an issue.

v-schhabra commented 1 month ago

Hi @directhex

  1. What kind of ARM service endpoint are you using in AzureFileCopy tasks?
  2. Where is the build agent running?
  3. How does the whole pipeline definition looks like?
  4. Share the complete debug logs?

Please share these details in order to investigate further.

@Schniefel Could you pls let us know if you are still facing this issue or not? If yes then pls do share the details.

mrlioncub commented 1 month ago

I upload big files. After every ~56m:

2024-09-25T05:25:23.9194509Z Task         : Azure file copy
2024-09-25T05:25:23.9194562Z Description  : Copy files to Azure Blob Storage or virtual machines
2024-09-25T05:25:23.9194630Z Version      : 6.245.1
...
2024-09-25T05:25:36.8413980Z ##[command] & "AzCopy\AzCopy.exe" copy "\\sqlbackups\Backup\diff\db_*" "https://xxx.blob.core.windows.net/sqlbackups/db"  --overwrite ifSourceNewer --log-level=DEBUG
2024-09-25T05:25:36.9402275Z INFO: Scanning...
2024-09-25T05:25:36.9404400Z INFO: AzCopy.exe 10.25.1: A newer version 10.26.0 is available to download
...
2024-09-25T06:20:42.5350255Z 89.7 %, 4 Done, 0 Failed, 1 Pending, 1 Skipped, 6 Total, 2-sec Throughput (Mb/s): 1107.1694 (Disk may be limiting speed)
2024-09-25T06:20:42.9203413Z ##[error]Exit code 2 returned from process: file name 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', arguments '-NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". ([scriptblock]::Create('if ([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) { [Console]::InputEncoding = New-Object Text.UTF8Encoding $false } if (!$PSHOME) { $null = Get-Item -LiteralPath ''variable:PSHOME'' } else { Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Management\Microsoft.PowerShell.Management.psd1'')) ; Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psd1'')) }')) 2>&1 | ForEach-Object { Write-Verbose $_.Exception.Message -Verbose } ; Import-Module -Name 'C:\azagent\A1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.245.1\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1' -ArgumentList @{ NonInteractive = $true } -ErrorAction Stop ; $VerbosePreference = 'SilentlyContinue' ; $DebugPreference = 'SilentlyContinue' ; Invoke-VstsTaskScript -ScriptBlock ([scriptblock]::Create('. ''C:\azagent\A1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.245.1\AzureFileCopy.ps1'''))"'.
Rudresh-Infy commented 5 hours ago

@directhex , @onetocny By any chance do you guys have any solution to this? Because I am also facing the same issue when running my release pipeline via 1ES.