Azure Service Connection using certificate authentication fails to get cert file password

AngleOSaxon commented 1 year ago

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name:

Any that take an AzureServiceConnection argument

Environment

Server - Azure Pipelines or TFS on-premises?
- Azure Pipelines

Account Name: transcardscm Team Project Name: AccessControl Build Definition Name/Build number: AccessControl Site - UAT - Release, build number 20221216.15

Agent - Private:
- If using private agent, provide the OS of the machine running the agent and the agent version:

OS: Windows Server 2016 Datacenter, version 1607, build 14393.5582 Agent Version: 2.213.2

Issue Description

I am using a privately-hosted agent and an Azure Service Connection configured with certificate authentication. When it attempts to execute a pipeline that authenticates to Azure using this service connection, it fails with this error:

"D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Can't open file "D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Error getting passwords

These pipelines and Service Connections work flawlessly on Microsoft-hosted agents.

Examining the private build agent with Procmon shows that openssl is handling the supplied password file path incorrectly, treating it as a relative file name rather than an absolute path.

In the task scripting, the password file path appears to be surrounded in quotes by line 257 in the Utility.ps1 file. Removing those quote marks allows the task to successfully read the password file and certificate, and to successfully authenticate.

Although removing the quotes does potentially allow spaces in the file path to break the command, the other paths supplied to the same command are in the same directory and are not quoted either, so this seems unlikely to cause any new issues.

Is there a known configuration change I can make to my private agent host so that openssl will process the $pfxPasswordFilePath properly, the way it does on the Microsoft-hosted agents? Or should I open a PR to remove the quotes from that argument in Utility.ps1?

Task logs

PrivateHost_FailedAuth_Cert.zip

Error logs

VERBOSE: Entering Invoke-VstsTool.
VERBOSE:  FileName: 'D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe'
VERBOSE:  Arguments: 'pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"'
VERBOSE:  RequireExitCodeZero: 'True'
"D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Can't open file "D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Error getting passwords
VERBOSE: Exit code: 1
VERBOSE: Leaving Invoke-VstsTool.
##[error]Process 'openssl.exe' exited with code '1'.
##[debug]Processed: ##vso[task.logissue type=error]Process 'openssl.exe' exited with code '1'.
VERBOSE: Leaving Initialize-AzModule.

peterhut commented 1 year ago

I had the same issue on 1 server, but not on another. The key difference ended up being the PowerShell version: using 7.3.2 gives the error, but PowerShell version 7.2.9 does not. Therefore my workaround was installing 7.2.9 on both servers.

jberezanski-mdg commented 1 year ago

I have the same problem. In my case: AzurePowerShell task 5.225.1, PowerShell version 7.3.5. Downgrading PowerShell is not an option for my team.

==============================================================================
Task         : Azure PowerShell
Description  : Run a PowerShell script within an Azure environment
Version      : 5.225.1
Author       : Microsoft Corporation
Help         : https://aka.ms/azurepowershelltroubleshooting
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\B\2\_temp\8b3411ed-3826-4e75-98d1-4958ce54e739.ps1'"
Added TLS 1.2 in session.
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\2.12.1\Az.Accounts.psd1 -Global
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Clear-AzConfig -DefaultSubscriptionForLogin
"D:\B\2\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.225.1\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\B\2\_temp\clientcertificate.pem -out D:\B\2\_temp\clientcertificate.pfx -password file:"D:\B\2\_temp\clientcertificatepassword.txt"
Can't open file "D:\B\2\_temp\clientcertificatepassword.txt"
Error getting passwords
##[error]Process 'openssl.exe' exited with code '1'.
##[error]There was an error with the service principal used for the deployment.
##[error]PowerShell exited with code '1'.
Disconnect-AzAccount -Scope Process -ErrorAction Stop
Clear-AzContext -Scope Process -ErrorAction Stop

KrazeyKami commented 11 months ago

Same issue. Can't downgrade.

jberezanski-mdg commented 11 months ago

The immediate cause of the problem seems to be a change in the way Invoke-Expression quotes command lines passed to external executables, which apparently happened between PS 7.2 and 7.3.

The AzurePowerShell task calls Initialize-AzModule from VstsAzureHelpers_ and execution flow eventually enters the ConvertTo-Pfx function. There a command line for openssl is constructed and passed to Invoke-VstsTool from VstsTaskSdk:

$openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:`"$pfxPasswordFilePath`""

Invoke-VstsTool -FileName $openSSLExePath -Arguments $openSSLArgs -RequireExitCodeZero

Invoke-VstsTool passes its input directly to Invoke-Expression:

Invoke-Expression "& '$FileName' --% $Arguments"

Here are the actual command lines, as observed via Sysinternals Process Monitor, on various PowerShell versions, with System_DefaultWorkingDirectory set to a path with and without spaces:

PS 5.1

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password file:"F:\Temp\nospaces\clientcertificatepassword.txt"

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password file:"F:\Temp\with spaces\clientcertificatepassword.txt"

PS 7.2.12

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password file:"F:\Temp\nospaces\clientcertificatepassword.txt"

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password file:"F:\Temp\with spaces\clientcertificatepassword.txt"

PS 7.3.6

"F:\Temp\openssl\openssl.exe" pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password "file:\"F:\Temp\nospaces\clientcertificatepassword.txt\""

"F:\Temp\openssl\openssl.exe" pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password "file:\"F:\Temp\with" "spaces\clientcertificatepassword.txt\""

Two things are apparent here: 1) Quotes embedded in the command line are doubly-quoted on PS 7.3. The command line is even more mangled when the path contains a space. 2) The code does not attempt to handle spaces in the values of the -in and -out arguments to openssl. On all PowerShell versions, the code does not work correctly if $Env:System_DefaultWorkingDirectory (or $Env:Agent_TempDirectory) contains spaces.

Fully fixing the problem with spaces in paths seems difficult to do, due to the design of Invoke-VstsTool and its reliance on Invoke-Expression and a single parameters string (instead of an array of parameters). However, the regression on PS 7.3 can be fixed simply by removing the quotes from the -password argument. This will make the behavior on PS 7.3 not worse than on earlier PowerShell versions - paths with spaces will still not work, but paths without them will (and I suspect this is the typical case for agent working directories).

Specifically, I propose the following change:

diff --git a/Tasks/Common/VstsAzureHelpers_/Utility.ps1 b/Tasks/Common/VstsAzureHelpers_/Utility.ps1
index 3b49b5c2c9..ef482fedee 100644
--- a/Tasks/Common/VstsAzureHelpers_/Utility.ps1
+++ b/Tasks/Common/VstsAzureHelpers_/Utility.ps1
@@ -355,7 +355,7 @@ function ConvertTo-Pfx {
     $env:OPENSSL_CONF = "$PSScriptRoot\openssl\openssl.cnf"
     $env:RANDFILE=".rnd"

-    $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:`"$pfxPasswordFilePath`""
+    $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:$pfxPasswordFilePath"

which will result in the same command line being generated in all three PS versions, fixing this issue. Would this be acceptable?

andreascaesar commented 9 months ago

Any progress on this? Still the same issue in 7.4...

btull89 commented 6 months ago

This has become a problem in several of my azure pipelines over the weekend. Can Microsoft provide an update to this issue? We are using Azure DevOps Hosted Agents.

AngleOSaxon commented 6 months ago

As a temporary workaround, they did provide scripts to downgrade Powershell back to 7.2.17 when they announced the update. I added them as steps at the start of my pipelines and they successfully downgraded the Powershell version used by all subsequent steps and got my pipelines running again.

It does need to get fixed, though; staying on 7.2 forever isn't a great plan.

Jetelaczek commented 6 months ago

downgrade

Morning, could you please give me some example? Facing same issue, but don't understand how you 'added them as steps at the start of my pipelines'.

Thanks!

AngleOSaxon commented 6 months ago

Sure. I created a script file called downgrade-powershell.cmd and placed it in a scripts directory, and then added this step early in each pipeline:

  - task: CmdLine@2
    displayName: Downgrade to Powershell 7.2
    inputs:
      failOnStderr: true
      workingDirectory: $(Agent.BuildDirectory)/scripts
      script: downgrade-powershell.cmd

the contents of downgrade-powershell.cmd are

set "extractPath=C:\Program Files\PowerShell\7"
curl -sLO https://github.com/PowerShell/PowerShell/releases/download/v7.2.17/PowerShell-7.2.17-win-x64.zip
RMDIR "%extractPath%" /S /Q
7z x PowerShell-7.2.17-win-x64.zip -o"%extractPath%"
pwsh --version

My pipelines almost all run on Windows, so I'm able to get away with just using a CmdTask and a batch file. If you've got a more heterogeneous environment, you might need to also add tasks with the macOS and Linux downgrade scripts they provided. You should be able to pick which one to run with something like this:

  - ${{ if eq(Agent.OS, 'Windows_NT') }}:
    - script: windows-downgrade-powershell.cmd
  - ${{ elseif eq(Agent.OS, 'Darwin') }}:
    - script: macos-downgrade-powershell.sh
  - ${{ elseif eq(Agent.OS, 'Linux') }}:
    - script: linux-downgrade-powershell.sh

(disclaimer: not tested)

quality-leftovers commented 6 months ago

Our pipelines are broken now, too. It would be nice if either

azure devops pipeline tasks didn't randomly break so often
microsoft would actually try to fix stuff asap when stuff breaks instead of being so passive

Jetelaczek commented 6 months ago

I had a chat with Microsoft support and what works for our pipelines is to use previous version of MS hosted agent (use 'windows-2019' instead of 'windows-latest') and use 'AzurePowerShell@4'...

Jetelaczek commented 6 months ago

Thanks much!

Tomas J.

From: Andrew Bikle @.> Sent: Thursday, February 1, 2024 3:06 PM To: microsoft/azure-pipelines-tasks @.> Cc: Tomáš Jetelina @.>; Comment @.> Subject: Re: [microsoft/azure-pipelines-tasks] Azure Service Connection using certificate authentication fails to get cert file password on private agent (Issue #17581)

Sure. I created a script file called downgrade-powershell.cmd and placed it in a scripts directory, and then added this step early in each pipeline:

task: @.***

displayName: Downgrade to Powershell 7.2

inputs:

failOnStderr: true

workingDirectory: $(Agent.BuildDirectory)/scripts

script: downgrade-powershell.cmd

the contents of downgrade-powershell.cmd are

set "extractPath=C:\Program Files\PowerShell\7"

curl -sLO https://github.com/PowerShell/PowerShell/releases/download/v7.2.17/PowerShell-7.2.17-win-x64.zip

RMDIR "%extractPath%" /S /Q

7z x PowerShell-7.2.17-win-x64.zip -o"%extractPath%"

pwsh --version

My pipelines almost all run on Windows, so I'm able to get away with just using a CmdTask and a batch file. If you've got a more heterogeneous environment, you might need to also add tasks with the macOS and Linux downgrade scripts they provided. You should be able to pick which one to run with something like this:

${{ if eq(Agent.OS, 'Windows_NT') }}:
- script: windows-downgrade-powershell.cmd
${{ elseif eq(Agent.OS, 'Darwin') }}:
- script: macos-downgrade-powershell.sh
${{ elseif eq(Agent.OS, 'Linux') }}:
- script: linux-downgrade-powershell.sh

(disclaimer: not tested)

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/17581#issuecomment-1921406601, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHI3LVGBRVGGCEZI6H6PQZDYROONHAVCNFSM6AAAAAATVRCPI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDMNRQGE. You are receiving this because you commented.Message ID: @.**@.>>

lenapera commented 6 months ago

We have the same issue in our pipelines with the Azure DevOps hosted agents. This affects productive deployments and downgrading isn't a permanent solution.

quality-leftovers commented 5 months ago

Can you please provide an update on this?

AngleOSaxon commented 5 months ago

I have no updates to provide. I've created GH-19558, a pull request with a change that I believe solves the problem. It needs to be reviewed by someone at Microsoft, but I do not have a way to cause that to happen.

AngleOSaxon commented 5 months ago

It looks as though a change just went in that will resolve the issue. Hopefully a new version incorporating it will be released in the next month or so.

AngleOSaxon commented 3 months ago

My pipeline agents just updated to version 3.238.0, which incorporates the fix for this bug. I was able to successfully run pipelines using certificate authentication with Powershell 7.4, so I believe this issue is resolved.

If your pipelines are still encountering this issue, make sure they're updated to the latest version.

microsoft / azure-pipelines-tasks