[BUG]: SSL handshake error with some Azure CLI commands using the PowerShell@2 core

pumaloso commented 3 months ago

What happened?

After upgrading agent to latest version (3.241.0) the following error SSL errors started to appear when using the PowerShell@2 core task:

ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006).
Certificate verification failed. This typically happens when using Azure CLI behind a proxy that intercepts traffic with a self-signed certificate. Please add this certificate to the trusted CA bundle. More info: https://docs.microsoft.com/cli/azure/use-cli-effectively#work-behind-a-proxy.

Tried to fix it with well-known suggestions without success.

Versions

Azure DevOps version 3.241.0 / Windows 10

Environment type (Please select at least one enviroment where you face this issue)

[x] Self-Hosted
[ ] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

Starting: PowerShell
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.239.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent\_work\_temp\d8fc052a-9322-4a41-b081-32b75e92b389.ps1'"
ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)
Certificate verification failed. This typically happens when using Azure CLI behind a proxy that intercepts traffic with a self-signed certificate. Please add this certificate to the trusted CA bundle. More info: https://docs.microsoft.com/cli/azure/use-cli-effectively#work-behind-a-proxy.
ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)
Certificate verification failed. This typically happens when using Azure CLI behind a proxy that intercepts traffic with a self-signed certificate. Please add this certificate to the trusted CA bundle. More info: https://docs.microsoft.com/cli/azure/use-cli-effectively#work-behind-a-proxy.
ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)
Certificate verification failed. This typically happens when using Azure CLI behind a proxy that intercepts traffic with a self-signed certificate. Please add this certificate to the trusted CA bundle. More info: https://docs.microsoft.com/cli/azure/use-cli-effectively#work-behind-a-proxy.
FATAL ERROR: failed to insert data into the storage table
At C:\agent\_work\1\s\src\powershell\AzureTable.ps1:57 char:13
+             throw "FATAL ERROR: failed to insert data into the storag …
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : OperationStopped: (FATAL ERROR: failed…o the storage table:String) [], RuntimeException
+ FullyQualifiedErrorId : FATAL ERROR: failed to insert data into the storage table
##[error]PowerShell exited with code '1'.
Finishing: PowerShell

pumaloso commented 3 months ago

Last known working version: 3.232.4

ivanduplenskikh commented 1 month ago

@pumaloso, thanks for reporting! We are working on more prioritized issues at the moment, but will get back to this one soon. Could you please provide some detailed information about your pipeline? Do you use Python in your pipeline? Could you provide a minimal example to reproduce this problem?

DmitriiBobreshev commented 1 month ago

Hi @pumaloso, after the investigation, I was able to reproduce the error on the latest agent as well as on the oldest ones(including 3.232.4). From what I see the problem with "self-signed certificate in certificate chain" not related to the azure-pipelines-agent because the agent itself can communicate with AzDevOps and third-party services through the self-signed certificate when the sslskipcertvalidation option passed during the configuration. The Powershell task is just a wrapper for other tools and can't control proxy behaviour of the task. Most probably something is wrong on your machine or with AzureCLI, one of the option is that the certificate was changed or removed from trusted certificates. Another option is that the Azure CLI version was changed and the flags like AZURE_CLI_DISABLE_CONNECTION_VERIFICATION stopped working. I believe you need to check installed tools from the last worked run and the failed one. Also, could you please create an issue in azure-cli repo: https://github.com/Azure/azure-cli?

Also, if we're speaking about the error from log output section, could you try to renew your certifcate and check it again as it might be a problem.

DmitriiBobreshev commented 1 month ago

@pumaloso, I'm closing the WI, please feel free to reopen it if you have any questions

microsoft / azure-pipelines-agent