search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.48k stars 2.61k forks source link

[BUG]: AzureKeyVault@2 is failing on the Node 20 version. #20462

Open youngercloud opened 4 days ago

youngercloud commented 4 days ago

New issue checklist

Task name

AzureKeyVault@2

Task version

2.244.1

Issue Description

Task is failing with network_error, status message: Network request failed when requesting the token to access the KeyVault https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token.

Observations:

  1. This issue only occurs in China. However, a direct curl command with https://login.microsoftonline.com works well, so I believe this is not an actual network issue.
  2. This issue does not happen with version 2.244.0. The current code only specifies AzureKeyVault@2, so the version 2.244.0 was found in the latest successful run. The version switch from 2.244.0 to 2.244.1 happened automatically last week.
  3. I've tried to use the code AzureKeyVault@2.244.0 in the Azure pipelines. However, the patch version number seems to be out of my control, and the pipeline is still using version 2.244.1.

Updated Observation:

  1. Noticed that the machine has a proxy enabled. Turning off the proxy temporart can help to resolve this issue.

Therefore, may I know what is the difference between the Default version and the Node 20 version, and how this version change occurred? Understanding this might help resolve the issue. Thanks!

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 20.04.2 LTS (Docker: Ubuntu 22.04.4 LTS with agent version 3.244.1)

Relevant log output

SubscriptionId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
Key vault name: xx-xxxxxxxx.
Downloading secret value for: samplesecret1.
Downloading secret value for: samplesecret2.
##[error]
samplesecret1: "Could not fetch access token for Azure. Status code: network_error, status message: Network request failed"
samplesecret2: "Could not fetch access token for Azure. Status code: network_error, status message: Network request failed"

Full task logs with system.debug enabled

##[debug][Tue, 24 Sep 2024 13:02:13 GMT] : [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] : @azure/msal-node@2.10.0 : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Tue, 24 Sep 2024 13:02:13 GMT] : [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] : @azure/msal-common@14.13.0 : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token
##[debug]Agent environment resources - Disk: / Available 471922.32 MB out of 503837.41 MB, Memory: Used 851.00 MB out of 15744.00 MB, CPU: Usage NaN%
##[debug]MSAL - retrying getMSALToken - temporary error code: network_error
##[debug]MSAL - retrying getMSALToken - remaining attempts: 1
##[debug]MSAL - getMSALToken called. force=undefined
##[debug][Tue, 24 Sep 2024 13:02:17 GMT] : [] : @azure/msal-node@2.10.0 : Info - acquireTokenByClientCredential called
##[debug][Tue, 24 Sep 2024 13:02:17 GMT] : [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] : @azure/msal-node@2.10.0 : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Tue, 24 Sep 2024 13:02:17 GMT] : [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] : @azure/msal-common@14.13.0 : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token
##[debug]Agent environment resources - Disk: / Available 471922.32 MB out of 503837.41 MB, Memory: Used 851.00 MB out of 15744.00 MB, CPU: Usage NaN%
##[debug]{}
##[debug]Downloaded part [0 - 2] (took 28146 ms) (total 2 secrets)
##[debug]task result: Failed
##[error]

Repro steps

- task: AzureKeyVault@2
  displayName: 'Get secret'
  inputs:
    azureSubscription: 'xx-xxxxxx-vaults'
    KeyVaultName: xx-xxxxxxxx
    SecretsFilter: |
      samplesecret1,
      samplesecret2
v-schhabra commented 3 days ago

Hi @youngercloud Thanks for reporting this issue. We are checking on it.

youngercloud commented 2 days ago

@v-schhabra Thanks so much! By the way, for the fact that users can specify major and minor versions but cannot specify patch versions for a task, is intentional or should it be considered a bug? Thanks!

v-schhabra commented 2 days ago

Hi @youngercloud Users can specify the major, minor and patch versions of the task in the yaml pipeline. Make sure that the version should be there in the prod. Could you please share the task version which you are not able to specify?

youngercloud commented 1 day ago

@v-schhabra Thanks! Just wanted to share the following scenario is happening to me: When I try to specify the version AzureKeyVault@2.244.0, the pipeline redirects to version AzureKeyVault@2.244.1 Similarly, when I try to specify version AzureKeyVault@2.243.0, it also redirects to AzureKeyVault@2.243.1.

By the way, I noticed that version 2.246.1 has been updated. This version also failed, just like 2.244.1. The last successful version is 2.244.0.

The major difference between 2.244.0 and 2.244.1 is that I found msalv1 and msalv2 in 2.244.1's node_modules, and they are used in the authorization process of login.microsoft.com. Currently, I am investigating in this NodeJS package and trying to figure out the reason why it cause the failure of AzureKeyVaultV2.

Best,

Deekshitha981 commented 1 day ago

Hi, Fix has been rolled out. https://github.com/microsoft/azure-pipelines-tasks/pull/20470

youngercloud commented 1 day ago

@Deekshitha981 Thank you! However, version 2.246.1 still has the same issue...

Also, when I tried specifying - task: AzureKeyVault@2.246.0 in my code, I still noticed the following log in my pipeline run.

==============================================================================
Task         : Azure Key Vault
Description  : Download Azure Key Vault secrets
Version      : 2.246.1

That's the third observation I mentioned in the issue.

youngercloud commented 1 day ago

@Deekshitha981 More information that I know,

The succeed version AzureKeyVault@2.244.0 is using Node 16, @azure/msal-common@14.14.0 and @azure/msal-node@2.12.0.

The current version AzureKeyVault@2.246.1 us using Node 20, @azure/msal-common@14.13.0 and @azure/msal-node@2.10.0.

Also, to explain what I meant in the previous post.

agent@azuredevops:/azp/_work/_tasks/AzureKeyVault_1e244d32-2dd4-4165-96fb-b7441ca9331e$ ls 2.244.0/node_modules/msalv1
ls: cannot access '2.244.0/node_modules/msalv1': No such file or directory

agent@azuredevops:/azp/_work/_tasks/AzureKeyVault_1e244d32-2dd4-4165-96fb-b7441ca9331e$ ls 2.244.0/node_modules/msalv2
ls: cannot access '2.244.0/node_modules/msalv2': No such file or directory

agent@azuredevops:/azp/_work/_tasks/AzureKeyVault_1e244d32-2dd4-4165-96fb-b7441ca9331e$ ls 2.246.1/node_modules/msalv1
LICENSE  README.md  dist  node_modules  package.json

agent@azuredevops:/azp/_work/_tasks/AzureKeyVault_1e244d32-2dd4-4165-96fb-b7441ca9331e$ ls 2.246.1/node_modules/msalv2
LICENSE  README.md  dist  node_modules  package.json  src
Deekshitha981 commented 1 day ago

Hi @youngercloud, Could you please provide the console logs by setting system.debug to true with current version.