[REGRESSION]: npmAuthenticate silently fails authenticate to Azure Devops npm feeds

sgtwilko commented 2 months ago

New issue checklist

[X] I searched for existing GitHub issues
[X] I read pipeline troubleshooting guide
[X] I checked how to collect logs

Task name

npmAuthenticate

Breaking task version

0.244.1

Last working task version

0.238.3

Regression Description

Although the task appears to work, when using npm ci packages being installed from our org scoped feed fail to install due to 401 authentication failures.

When reverting to version 0.238.3 of the npmAuthenticate task the subsequent npm ci task completes successfully.

Environment type (Please select at least one enviroment where you face this issue)

[ ] Self-Hosted
[X] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows-2019

Relevant log output

##[debug]Getting credentials for local feeds
##[debug]SYSTEMVSSCONNECTION exists true
##[debug]Got auth token, setting it as secret so it does not print in console log
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Getting credentials for local feeds
##[debug]SYSTEMVSSCONNECTION exists true
##[debug]Got auth token, setting it as secret so it does not print in console log
##[debug]Processed: ##vso[task.setsecret]***
Adding credentials for a local registry
##[debug]Adding auth for registry: [****URL Redacted****]
##[debug]Successfully appended .npmrc
##[debug]Successfully pushed .npmrc
Adding credentials for a local registry
##[debug]Adding auth for registry: [****URL Redacted****]
##[debug]Successfully appended .npmrc
##[debug]Successfully pushed .npmrc
Adding credentials for a local registry
##[debug]Adding auth for registry: [****URL Redacted****]
##[debug]Successfully appended .npmrc
##[debug]Successfully pushed .npmrc
##[debug]Agent.Version=3.243.1
##[debug]SYSTEM_TASKINSTANCEID=[****Redacted****]
##[debug]SYSTEM_JOBID=[****Redacted****]
##[debug]SYSTEM_PLANID=[****Redacted****]
##[debug]SYSTEM_COLLECTIONID=[****Redacted****]
##[debug]SYSTEM_PULLREQUEST_ISFORK=False
##[debug]AGENT_ID=191
##[debug]AGENT_MACHINENAME=fv-az787-174
##[debug]AGENT_NAME=Azure Pipelines 10
##[debug]AGENT_JOBSTATUS=Succeeded
##[debug]AGENT_OS=Windows_NT
##[debug]AGENT_OSARCHITECTURE=X64
##[debug]AGENT_VERSION=3.243.1
##[debug]BUILD_BUILDID=1430786
##[debug]BUILD_BUILDNUMBER=20240828.10
##[debug]BUILD_BUILDURI=vstfs:///Build/Build/1430786
##[debug]BUILD_CONTAINERID=46611300
##[debug]BUILD_DEFINITIONNAME=[****Redacted****]
##[debug]BUILD_DEFINITIONVERSION=121
##[debug]BUILD_REASON=Manual
##[debug]Processed: ##vso[telemetry.publish area=Packaging;feature=NpmAuthenticateV0][****Redacted****]
Finishing: npmAuthenticate

##[debug]Evaluating condition for step: 'npm ci'
##[debug]Evaluating: ne(variables['NPM_CACHE_RESTORED'], 'true')
##[debug]Evaluating ne:
##[debug]..Evaluating indexer:
##[debug]....Evaluating variables:
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'NPM_CACHE_RESTORED'
##[debug]..=> 'false'
##[debug]..Evaluating String:
##[debug]..=> 'true'
##[debug]=> True
##[debug]Expanded: ne('false', 'true')
##[debug]Result: True
Starting: npm ci
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.244.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
##[debug]VstsTaskSdk 0.20.1 commit ab885244acac2b6da000c01e13bcbcb47eb3e76d
##[debug]Entering D:\a\_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.244.3\cmdline.ps1.
##[debug]Loading resource strings from: D:\a\_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.244.3\task.json
##[debug]Loaded 6 strings.
##[debug]SYSTEM_CULTURE: 'en-US'
##[debug]Loading resource strings from: D:\a\_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.244.3\Strings\resources.resjson\en-US\resources.resjson
##[debug]Loaded 6 strings.
##[debug]INPUT_FAILONSTDERR: 'false'
##[debug] Converted to bool: False
##[debug]INPUT_SCRIPT: 'npm ci'
##[debug]INPUT_WORKINGDIRECTORY: 'D:\a\1\s\frontend\'
##[debug]Asserting container path exists: 'D:\a\1\s\frontend\'
Generating script.
Script contents: shell
npm ci
##[debug]AGENT_VERSION: '3.243.1'
##[debug]AGENT_TEMPDIRECTORY: 'D:\a\_temp'
##[debug]Asserting container path exists: 'D:\a\_temp'
##[debug]Asserting leaf path exists: 'C:\Windows\system32\cmd.exe'
========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\283259c2-c0d6-42aa-9b8f-96460debec60.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'

Full task logs with system.debug enabled

See above:

UNSUCCESSFUL RUN

 [REPLACE THIS WITH YOUR INFORMATION]

SUCCESSFUL RUN

 [REPLACE THIS WITH YOUR INFORMATION]

Repro steps

# npmAuthenticate appears to work, but then `npm ci` will then fail.
  - task: npmAuthenticate
    inputs:
      workingFile: './frontend/.npmrc'

  - script: |
      npm ci
    displayName: 'npm ci'
    workingDirectory: frontend/

decayedCell commented 2 months ago

Similar issue here:

Breaking task version

0.244.3

Last working task version

0.238.3

Relevant log output

    ##[debug]Evaluating condition for step: '{Redacted} - npm Authenticate'
    ##[debug]Evaluating: SucceededNode()
    ##[debug]Evaluating SucceededNode:
    ##[debug]=> True
    ##[debug]Result: True
    Starting: {Redacted} - npm Authenticate
    ==============================================================================
    Task         : npm authenticate (for task runners)
    Description  : Don't use this task if you're also using the npm task. Provides npm credentials to an .npmrc file in your repository for the scope of the build. This enables npm task runners like gulp and Grunt to authenticate with private registries.
    Version      : 0.244.1
    Author       : Microsoft Corporation
    Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/npm-authenticate
    ==============================================================================
    ##[debug]Using node path: /agent/externals/node20_1/bin/node
    ##[debug]agent.TempDirectory=/datadrive/agent/_temp
    ##[debug]loading inputs and endpoints
    ##[debug]loading INPUT_WORKINGFILE
    ##[debug]loading INPUT_WORKLOADIDENTITYSERVICECONNECTION
    ##[debug]loading INPUT_FEEDURL
    ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
    ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
    ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
    ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
    ##[debug]loaded 7
    ##[debug]Agent.ProxyUrl=undefined
    ##[debug]Agent.CAInfo=undefined
    ##[debug]Agent.ClientCert=undefined
    ##[debug]Agent.SkipCertValidation=undefined
    ##[debug]check path : {Redacted}/module.json
    ##[debug]adding resource file: {Redacted}/module.json
    ##[debug]system.culture=en-US
    ##[debug]check path : {Redacted}/task.json
    ##[debug]adding resource file: {Redacted}/task.json
    ##[debug]system.culture=en-US
    ##[debug]workingFile={Redacted}/.npmrc
    ##[debug]EXISTING_ENDPOINTS=undefined
    Adding authentication to the .npmrc file at {Redacted}/.npmrc
    ##[debug]SAVE_NPMRC_PATH=undefined
    ##[debug]Agent.BuildDirectory={Redacted}/22
    ##[debug]testing directory '{Redacted}/22/npmAuthenticate'
    ##[debug]testing directory '{Redacted}/22'
    ##[debug]mkdir '{Redacted}/npmAuthenticate'

    ##[debug]check path : {Redacted}/task.json
    ##[debug]adding resource file: {Redacted}/task.json
    ##[debug]system.culture=en-US
    ##[debug]failOnStderr=false
    ##[debug]script=npm install {Redacted}
    ##[debug]workingDirectory={Redacted}
    ##[debug]check path : {Redacted}
    Generating script.
    ##[debug]Agent.Version=3.243.0
    ##[debug]agent.tempDirectory=/datadrive/agent/_temp
    ##[debug]check path : /datadrive/agent/_temp
    Script contents:
    npm install {Redacted}
    ========================== Starting Command Output ===========================
    ##[debug]which 'bash'
    ##[debug]found: '/usr/bin/bash'
    ##[debug]which '/usr/bin/bash'
    ##[debug]found: '/usr/bin/bash'
    ##[debug]/usr/bin/bash arg: --noprofile
    ##[debug]/usr/bin/bash arg: --norc
    ##[debug]/usr/bin/bash arg: {Redacted}
    ##[debug]exec tool: /usr/bin/bash
    ##[debug]arguments:
    ##[debug]   --noprofile
    ##[debug]   --norc
    ##[debug]  {Redacted}
    /usr/bin/bash --noprofile --norc {Redacted}
    npm ERR! code E401
    npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/{Redacted}, Basic realm="https://pkgsproduks1.pkgs.visualstudio.com/", TFS-Federated

    npm ERR! A complete log of this run can be found in:
    npm ERR!     /home/AzDevOps/.npm/_logs/{Redacted}-debug.log

    ##[debug]Exit code 1 received from tool '/usr/bin/bash'
    ##[debug]STDIO streams have closed for tool '/usr/bin/bash'
    ##[error]Bash exited with code '1'.
    ##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;correlationId={Redacted};]Bash exited with code '1'.
    ##[debug]task result: Failed
    ##[debug]Processed: ##vso[task.complete result=Failed;done=true;]
    Finishing: {Redacted} - installing {Redacted}

mklenbw commented 2 months ago

Same here. Also reverted to 0.238.3.

sgtwilko commented 2 months ago

@embetten I see you've been working on this task, would you be able to say if this issue will be looked at soon?

AlexVTor commented 2 months ago

@sgtwilko The issue is being looked into, however I haven't been able to get a repro on my end (using the affected version). Can you provide some more information? I'm interested in:

Agent Version
Agent OS
Agent Type: Self hosted or Azure VM
Did you have other npm operations in the pipeline that also resulted in failures (publish, install, etc)?
From your original logs, can you confirm the .npmrc being authenticated is in the same working directory ('D:\a\1\s\frontend\') being used in the ci command?

TRStarr commented 1 month ago

@AlexVTor I am facing the same issue:

Version: 3.243.1
OS: Ubuntu 22.04.4 LTS
Type: Azure VM
Only fails on npm install - npm authenticate step happens before which does not fail - unable to get proceed to build step
logs show .npmrc service connection is being added and npm ci is has the working file .npmrc passed in
as @mklenbw states, reverting to 0.238.3 as authentication works

Is there a a way to upload the log files hidden from public view? as these contain sensitive information - happy to email over if required.

AlexVTor commented 1 month ago

@TRStarr Thanks for offering your help here. I don't think there's a way to upload files without them being available to others. I can think of two other approaches.

Frist, you can redact the sensitive information and post an .npmrc/logs here.
Secondly, we have an alternative support channel that supports the publishing of files away from public view. This would be the more secure route and would be greatly appreciated. If you choose to follow this route, the following site will let you sign in and report and issue: https://developercommunity.visualstudio.com/AzureDevOps.

Thanks again for your assistance

TRStarr commented 1 month ago

@AlexVTor Thank you for the quick reply - please find .npmrc and verbose logs files here

TRStarr commented 3 weeks ago

Hey @AlexVTor happy to confirm 0.246.3 is working all hunky dory! thank you very much for fixing this!!

AlexVTor commented 3 weeks ago

Thanks for the feedback! I've heard the same feedback on other fronts as well, so I will close this issue. If there are any more issues, please re-open or post here.

microsoft / azure-pipelines-tasks