[BUG]: Azure DevOps Inline Powershell Task not retrying on failure

aanhalzer commented 1 year ago

Task name

PowerShell

Task version

1.x

Environment type (Please select at least one enviroment where you face this issue)

[ ] Self-Hosted
[X] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows-2019

Task log

2023-07-09T16:05:24.1184239Z ##[section]Starting: Inline Powershell
2023-07-09T16:05:24.1381877Z ==============================================================================
2023-07-09T16:05:24.1382420Z Task         : Run Inline Powershell
2023-07-09T16:05:24.1382643Z Description  : Run the Powershell from a textbox
2023-07-09T16:05:24.1382836Z Version      : 1.1.7
2023-07-09T16:05:24.1383040Z Author       : Peter Groenewegen - Xpirit
2023-07-09T16:05:24.1385148Z Help         : Information on how to use the task: [Using the inline powershell vsts task](https://pgroene.wordpress.com/2016/05/20/using-the-inline-powershell-vsts-task)  
Tips and tricks:  
[Use VSTS variables](https://pgroene.wordpress.com/2017/12/07/use-vsts-variables-inline-powershell)  
  [Let your task fail](https://pgroene.wordpress.com/2017/12/08/let-your-inline-powershell-task-fail)  
  [Set progress](https://pgroene.wordpress.com/2017/12/11/4256)  
  [Change buildnumber](https://pgroene.wordpress.com/2017/12/12/set-buildnumber-from-inline-powershell)  
  [VSTS Powershell command overview](https://pgroene.wordpress.com/2017/12/13/vsts-powershell-commands)  
  [Call a WebHook](https://pgroene.wordpress.com/2017/12/14/call-a-webhook-from-inline-powershell)  
  [Download a file](https://pgroene.wordpress.com/2017/12/15/download-a-file-from-inline-powershell)  
  [Install a Powershell Module](https://pgroene.wordpress.com/2017/12/18/install-a-powershell-module-inline-powershell-vsts)  
  [Navigate VSTS as filesystem](https://pgroene.wordpress.com/2017/12/19/navigate-vsts-project-as-filesystem-from-inline-powershell)  
  [Make VSTS API Rest calls](https://pgroene.wordpress.com/2017/12/20/vsts-rest-api-from-inline-powershell)  
  [Script example: Act on failed build ](https://pgroene.wordpress.com/2017/12/21/let-your-build-work-for-you-shorten-the-feedback-loop)  

2023-07-09T16:05:24.1387395Z ==============================================================================
2023-07-09T16:05:25.6997287Z ##[command]& 'C:\Users\mssalesgmsaint$\AppData\Local\Temp\tmpA6D9.ps1' 
2023-07-09T16:05:25.7255540Z Before exception thrown
2023-07-09T16:05:25.9417822Z ##[error]This is an error.
2023-07-09T16:05:26.0025803Z ##[section]Finishing: Inline Powershell

Relevant log output

Does not apply. Please read "Additional Info"

Aditional info

I have a PowerShell Script task that needs to be configured for retries under failure, due to transient issues with the script. I noticed that in the first failure after configuring the retries, the retry was completely ignored.

In an effort to understand the problem I created a custom stage on my release with a simple Inline PowerShell task that is designed to fail immediately. This is the YAML, since I am not allowed to paste images:

steps:
- task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
  displayName: 'Inline Powershell'
  inputs:
    Script: |
     Write-Output 'Before exception thrown'
     throw "This is an error."
     Write-Output 'After exception thrown'
  retryCountOnTaskFailure: 3

No retries are ever attempted. This feature seems to either not apply for PowerShell tasks, or is broken. I have reviewed this document to deep detail: https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-195-update#automatic-retries-for-a-task and I am fully compliant with the guidance.

This all runs on a Deployment Group Job on agent version 2.194.0

JoeGaggler commented 1 year ago

Have you tried using the built-in PowerShell task? https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/powershell-v2?view=azure-pipelines

The task you are referencing in your YAML is not officially supported in the microsoft/azure-pipelines-tasks repo where you filed this issue.

ismayilov-ismayil commented 1 year ago

Hi @aanhalzer, thanks for reporting. This task (https://marketplace.visualstudio.com/items?itemName=petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell) is not supported by azure-pipeline-tasks team. Please try to contact with author directly.

aanhalzer commented 1 year ago

Thanks for the response. I do not believe this has anything to do with the developer that created that task. This is an issue I am seeing with first-party Microsoft owned tasks. Here an example I just tried:

YAML

steps:
- powershell: |
   Write-Output 'Before exception thrown'
   throw "This is an error."
   Write-Output 'After exception thrown'
  displayName: 'PowerShell Script'
  retryCountOnTaskFailure: 3

RELEASE LOGS

2023-07-10T19:58:53.5615674Z ##[section]Starting: PowerShell Script
2023-07-10T19:58:53.6257457Z ==============================================================================
2023-07-10T19:58:53.6258036Z Task         : PowerShell
2023-07-10T19:58:53.6258503Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2023-07-10T19:58:53.6258720Z Version      : 2.220.0
2023-07-10T19:58:53.6259032Z Author       : Microsoft Corporation
2023-07-10T19:58:53.6259409Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-07-10T19:58:53.6259739Z ==============================================================================
2023-07-10T19:58:55.7462448Z Generating script.
2023-07-10T19:58:55.7878846Z ========================== Starting Command Output ===========================
2023-07-10T19:58:55.8143637Z ##[command]"C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\azagent\A18\_work\_temp\b278b488-66a8-44ee-babe-59e7c2954a19.ps1'"
2023-07-10T19:58:56.2363259Z Before exception thrown
2023-07-10T19:58:56.3807398Z This is an error.
2023-07-10T19:58:56.3808678Z At C:\azagent\A18\_work\_temp\b278b488-66a8-44ee-babe-59e7c2954a19.ps1:5 char:1
2023-07-10T19:58:56.3809293Z + throw "This is an error."
2023-07-10T19:58:56.3809909Z + ~~~~~~~~~~~~~~~~~~~~~~~~~
2023-07-10T19:58:56.3810585Z     + CategoryInfo          : OperationStopped: (This is an error.:String) [], RuntimeException
2023-07-10T19:58:56.3811163Z     + FullyQualifiedErrorId : This is an error.
2023-07-10T19:58:56.3811613Z  
2023-07-10T19:58:56.4608104Z ##[error]PowerShell exited with code '1'.
2023-07-10T19:58:56.5026563Z ##[section]Finishing: PowerShell Script

What would your opinion be in this case?

JoeGaggler commented 1 year ago

That's odd, I just copied your powershell task and was able to see retry attempts:

2023-07-10T20:16:35.2035881Z ##[section]Starting: PowerShell Script
2023-07-10T20:16:35.2041066Z ==============================================================================
2023-07-10T20:16:35.2041208Z Task         : PowerShell
2023-07-10T20:16:35.2041258Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2023-07-10T20:16:35.2041335Z Version      : 2.220.0
2023-07-10T20:16:35.2041382Z Author       : Microsoft Corporation
2023-07-10T20:16:35.2041437Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-07-10T20:16:35.2041539Z ==============================================================================
2023-07-10T20:16:35.3612942Z Generating script.
2023-07-10T20:16:35.3657305Z ========================== Starting Command Output ===========================
2023-07-10T20:16:35.3675906Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/3bfcf5b8-d757-4a6f-bfd9-a8d2db6e1f22.ps1'
2023-07-10T20:16:37.1324431Z Before exception thrown
2023-07-10T20:16:37.3028292Z [31;1mException: [0m/home/vsts/work/_temp/3bfcf5b8-d757-4a6f-bfd9-a8d2db6e1f22.ps1:4
2023-07-10T20:16:37.3029072Z [36;1mLine |
2023-07-10T20:16:37.3029736Z [36;1m   4 | [0m [36;1mthrow "This is an error."[0m
2023-07-10T20:16:37.3030036Z [36;1m     | [31;1m ~~~~~~~~~~~~~~~~~~~~~~~~~
2023-07-10T20:16:37.3030240Z [31;1m[36;1m     | [31;1mThis is an error.
2023-07-10T20:16:37.3030420Z [0m
2023-07-10T20:16:37.3689095Z ##[error]PowerShell exited with code '1'.
2023-07-10T20:16:37.3720567Z ##[warning]RetryHelper encountered task failure, will retry (attempt #: 1 out of 3) after 1000 ms
2023-07-10T20:16:38.4887588Z Generating script.
2023-07-10T20:16:38.4895273Z ========================== Starting Command Output ===========================
2023-07-10T20:16:38.4929244Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/5514ecf9-87c2-427b-a39b-516bbc47766f.ps1'
2023-07-10T20:16:39.6490532Z Before exception thrown
2023-07-10T20:16:39.7667671Z [31;1mException: [0m/home/vsts/work/_temp/5514ecf9-87c2-427b-a39b-516bbc47766f.ps1:4
2023-07-10T20:16:39.7668718Z [36;1mLine |
2023-07-10T20:16:39.7669978Z [36;1m   4 | [0m [36;1mthrow "This is an error."[0m
2023-07-10T20:16:39.7671744Z [36;1m     | [31;1m ~~~~~~~~~~~~~~~~~~~~~~~~~
2023-07-10T20:16:39.7672412Z [31;1m[36;1m     | [31;1mThis is an error.
2023-07-10T20:16:39.7672864Z [0m
2023-07-10T20:16:39.8241672Z ##[error]PowerShell exited with code '1'.
2023-07-10T20:16:39.8296008Z ##[warning]RetryHelper encountered task failure, will retry (attempt #: 2 out of 3) after 4000 ms
2023-07-10T20:16:43.9502938Z Generating script.
2023-07-10T20:16:43.9540144Z ========================== Starting Command Output ===========================
2023-07-10T20:16:43.9545917Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/4a9e4a54-abc0-426a-a5d6-afd6ef369037.ps1'
2023-07-10T20:16:45.0811491Z Before exception thrown
2023-07-10T20:16:45.1980944Z [31;1mException: [0m/home/vsts/work/_temp/4a9e4a54-abc0-426a-a5d6-afd6ef369037.ps1:4
2023-07-10T20:16:45.1981544Z [36;1mLine |
2023-07-10T20:16:45.1981834Z [36;1m   4 | [0m [36;1mthrow "This is an error."[0m
2023-07-10T20:16:45.1982160Z [36;1m     | [31;1m ~~~~~~~~~~~~~~~~~~~~~~~~~
2023-07-10T20:16:45.1982448Z [31;1m[36;1m     | [31;1mThis is an error.
2023-07-10T20:16:45.1982665Z [0m
2023-07-10T20:16:45.2518118Z ##[error]PowerShell exited with code '1'.
2023-07-10T20:16:45.2577364Z ##[warning]RetryHelper encountered task failure, will retry (attempt #: 3 out of 3) after 9000 ms
2023-07-10T20:16:54.3686725Z Generating script.
2023-07-10T20:16:54.3694966Z ========================== Starting Command Output ===========================
2023-07-10T20:16:54.3728354Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/ce2cd3f7-0d11-41c4-8fed-bf806c2f8ce0.ps1'
2023-07-10T20:16:55.4915464Z Before exception thrown
2023-07-10T20:16:55.6064006Z [31;1mException: [0m/home/vsts/work/_temp/ce2cd3f7-0d11-41c4-8fed-bf806c2f8ce0.ps1:4
2023-07-10T20:16:55.6065294Z [36;1mLine |
2023-07-10T20:16:55.6066947Z [36;1m   4 | [0m [36;1mthrow "This is an error."[0m
2023-07-10T20:16:55.6067435Z [36;1m     | [31;1m ~~~~~~~~~~~~~~~~~~~~~~~~~
2023-07-10T20:16:55.6067795Z [31;1m[36;1m     | [31;1mThis is an error.
2023-07-10T20:16:55.6068204Z [0m
2023-07-10T20:16:55.6590365Z ##[error]PowerShell exited with code '1'.
2023-07-10T20:16:55.6634695Z ##[section]Finishing: PowerShell Script

aanhalzer commented 1 year ago

Oh! You have what I need! Can you share the task and deployment group configuration details? Or at least help me revising mine? My fear is that I am missing something radically obvious. I am attaching my setup in the form of screenshots to this comment:

I am using a Deployment Group Job. It has the correct version, per the documents. But perhaps there is some configuration that is blocking the retries?

aanhalzer commented 1 year ago

I have confirmed that retries work with an Agent Job. Is this an unsupported feature for Deployment Group Jobs? Or perhaps a faulty configuration?

aanhalzer commented 1 year ago

Upgraded build agent version, to no success

It seems this is wrongly documented as a supported feature for Deployment Group Jobs... https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-195-update#automatic-retries-for-a-task

Maybe "Not supported for agentless tasks." means that anything that is not an Agent Job lacks support.

JoeGaggler commented 1 year ago

Still works ok for me on a Deployment Group Job as well, though in my case I'm using only YAML and not the classic UI as you were showing in your screenshot. Here's my pipeline:

stages:
  - stage: pwtest
    displayName: pwtest
    dependsOn: []
    jobs:
      - deployment: pwtest
        dependsOn: []
        environment: 
          name: rd-bsquad
          resourceType: VirtualMachine
          tags: DevOpsPipeline
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: none
                - download: none
                - powershell: |
                    Write-Output 'Before exception thrown'
                    throw "This is an error."
                    Write-Output 'After exception thrown'
                  displayName: 'PowerShell Script'
                  retryCountOnTaskFailure: 3

JoeGaggler commented 1 year ago

I just tried this in a classic release, and was finally able to repro your issue. I tried flipping around the control options in different ways, but could not fix the issue. I agree with you that it's either a bug, or just insufficiently documented that classic releases with deployment group jobs do not get this particular feature.

I know changing your pipelines may not be feasible right now, but would still recommend the switch to YAML when you can manage it :-)

aanhalzer commented 1 year ago

I appreciate the help Joe. Changing to YAML is not a trivial task, but something we plan to do based on this situation. Closing the bug, thanks for your help!

microsoft / azure-pipelines-tasks