search

npm / cli

the package manager for JavaScript
https://docs.npmjs.com/cli/
Other
8.49k stars 3.17k forks source link

[BUG] npm 7 + Azure Devops Artifacts #2619

Open leepowelldev opened 3 years ago

leepowelldev commented 3 years ago

We have an issue very similar to this bug report https://github.com/npm/cli/issues/2183, which has apparently been fixed. However we're seeing the issue in the latest version of v7 (and previous versions).

Current Behavior:

We have a number of Azure Devops build pipelines which make use of the built in Artifacts (registry) feature. All build pipelines are already authenticated to retrieve these packages, and it works fine with v6.

The build pipelines utilise the npm task (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/npm?view=azure-devops) to perform the install. However the install task is failing with 401 errors.

An odd thing I have noticed in the logs is that the pipeline runs a npm config list and the results are different between using v6 and v7:

v6

SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
/opt/hostedtoolcache/node/14.15.4/x64/bin/npm --version
6.14.11
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
/opt/hostedtoolcache/node/14.15.4/x64/bin/npm config list
; cli configs
metrics-registry = "https://***.pkgs.visualstudio.com/_packaging/***-***-***-***-***/npm/registry/"
scope = ""
user-agent = "npm/6.14.11 node/v14.15.4 linux x64"

; environment configs
userconfig = "/home/vsts/work/1/npm/956552.npmrc"

; userconfig /home/vsts/work/1/npm/956552.npmrc
registry = "https://***.pkgs.visualstudio.com/_packaging/***-***-***-***-***/npm/registry/"

; node bin location = /opt/hostedtoolcache/node/14.15.4/x64/bin/node
; cwd = /path/to/App
; HOME = /home/vsts
; "npm config ls -l" to show all defaults.

/opt/hostedtoolcache/node/14.15.4/x64/bin/npm ci

v7

SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
/opt/hostedtoolcache/node/14.15.4/x64/bin/npm --version
7.5.2
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
/opt/hostedtoolcache/node/14.15.4/x64/bin/npm config list
; "user" config from /home/vsts/work/1/npm/956544.npmrc

//***.pkgs.visualstudio.com/_packaging/***-***-***-***-***/npm/registry/:_authToken = (protected) 
registry = "https://***.pkgs.visualstudio.com/_packaging/***-***-***-***-***/npm/registry/" 

; "env" config from environment

userconfig = "/home/vsts/work/1/npm/956544.npmrc" 

; "cli" config from command line options

omit = [] 
user-agent = "npm/7.5.2 node/v14.15.4 linux x64 ci/azure-pipelines" 

; node bin location = /opt/hostedtoolcache/node/14.15.4/x64/bin/node
; cwd = /path/to/App
; HOME = /home/vsts
; Run `npm config ls -l` to show all defaults.
/opt/hostedtoolcache/node/14.15.4/x64/bin/npm ci

As you can see, v7 shows this in the config //***.pkgs.visualstudio.com/_packaging/***-***-***-***-***/npm/registry/:_authToken = (protected), which v6 doesn't. And the install error log is around an invalid auth token. So not sure if this has something to do with it.

Install log Error:

4097 timing command:ci Completed in 22670ms
4098 verbose stack Error: Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/***-***-***-***-***, Basic realm="https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", TFS-Federated
4098 verbose stack     at /opt/hostedtoolcache/node/14.15.4/x64/lib/node_modules/npm/node_modules/npm-registry-fetch/check-response.js:113:17
4098 verbose stack     at processTicksAndRejections (internal/process/task_queues.js:93:5)
4099 verbose statusCode 401
4100 verbose pkgid yargs-parser@https://pkgs.dev.azure.com/****/_packaging/***/npm/registry/yargs-parser/-/yargs-parser-13.1.2.tgz
4101 verbose cwd /path/to/App
4102 verbose Linux 5.4.0-1036-azure
4103 verbose argv "/opt/hostedtoolcache/node/14.15.4/x64/bin/node" "/opt/hostedtoolcache/node/14.15.4/x64/bin/npm" "ci"
4104 verbose node v14.15.4
4105 verbose npm  v7.5.2
4106 error code E401
4107 error Unable to authenticate, your authentication token seems to be invalid.
4108 error To correct this please trying logging in again with:
4108 error     npm login
4109 verbose exit 1

Expected Behavior:

The install process should work as v6 does.

Environment:

Azure Devops npm task

leepowelldev commented 3 years ago

We've dug a bit further on this... the URI to our registry is:

https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/

However there is an alternate URI some of our developers are using that resolves to the same endpoint:

https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/

EDIT: This second URI is part of a URI move MS are in the process of rolling out (although it's been around for a while now) - https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/sep-10-azure-devops-launch#administration

This results in the package-lock.json file having the second URI as it's path to packages, while the .npmrc (created automatically by Azure) points the registry to the first URI.

As I said before, this wasn't an issue for v6, but does seem to have impacted v7 for the installs in the pipeline to fail.

leepowelldev commented 3 years ago

Might be related: https://github.com/npm/cli/issues/2508

BobBuildingCode commented 3 years ago

Thank you for reporting this, I thought I was going crazy when our builds suddenly started to fail. After downgrading NPM from 7.5.3 to 6.14.11, they started working again. So definitely something related to v7.

edit: upgraded to 7.5.4, same issue

leepowelldev commented 3 years ago

@BobBuildingCode i agree it’s an issue with v7. However from investigation enabling ‘dev.azure.com’ as the sole url in settings also seems to have an impact. But that might also mean you repointing your feed to that endpoint as it uses different urls. You can get them in feed settings.

leepowelldev commented 3 years ago

Update to the issue - There definitely seems to be something up with how v7 is handling the urls.

Our situation is a developer has his registry set to use https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/, he does an npm install which updates the package-lock.json file to point all the resolved values to this endpoint.

In the build pipeline we use the npm task in Azure to perform the install, which internally uses the https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/ registry. For some reason, npm v7 is attempting to use the resolved values in the package-json.lock file to fetch the packages which fails as the auth token being used internally is for https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/ yet our resolved urls are pointing to https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/.

I can replicate this locally, and have managed to find that if I add this to my npmrc it solves the issue depending on which registry my npmrc is pointing to...

//ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/:always-auth=true

or 

//pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:always-auth=true

I have tested this in v6 and it does not require this setting for either scenario and works as expected.

ljharb commented 3 years ago

For some reason

what other values should npm be using besides the ones in the lockfile?

leepowelldev commented 3 years ago

Yes you're right, it should use them - by 'some reason' I guess I mean it displays them in the verbose output when they fail. When it works (by adding always-auth) they end up at a different url 'https://fv1vsblobprodsu6weus82.blob.core.windows.net' - no idea why...

When I add the 'alway-auth' it works, even though always-auth is also set globally - maybe it's being ignored?

leepowelldev commented 3 years ago

@ljharb Is my understanding correct that if a registry is defined (either by an .npmrc file or cli flag) then NPM should transparently attempt to resolve packages from that registry rather than the urls in the package-lock.json file when installing?

ljharb commented 3 years ago

@leepowelldev that is my understanding as well.

leepowelldev commented 3 years ago

@ljharb That is what I think isn't happening - I don't think npm is respecting a defined registry (either by an .npmrc file or cli flag) and instead falling back to using the resolved url in the package-lock.json file (which may be a completely different registry with differnt authentication).

depascalis commented 3 years ago

I got it working by removing always-auth and regenerating access tokens.

Steps taken

This is on a Windows 10 machine, there are probably more ways to achieve the same:

  1. Verified that the organization used the new domain name URL dev.azure.com in DevOps > Organization Settings
    Switch existing organizations to use the new domain name URL
  2. Removed always-auth in the projects .npmrc (example is using scopes)
    C:\Repos\SomeProject\.npmrc 
    @ACCOUNT_NAME:registry=https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/
registry=https://registry.npmjs.org
  3. Revoked packages access tokens with "dev.azure.com" under Personal Settings in DevOps
  4. Deleted or commented out corresponding entries in the User/Global .npmrc
    C:\Users\My Windows User\.npmrc 
    ;//pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:username=VssSessionToken
;//pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:_password=1234ABCD123abc==
;//pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:email=not-used@example.com
  5. Refetched credentials
    Set up authentication on your development machine
    Command from project root 
    vsts-npm-auth -config .npmrc

Notes

The majority of the steps seems unnecessary, but it did not work by just removing always-auth. always-auth is never set anywhere in my npm configurations, but it looks like to be a default value from the output bellow?

Detailed output for comparison after taking the steps above 

PS C:\source\repos\SomeProject> vsts-npm-auth -config .npmrc -V Detailed

vsts-npm-auth v0.41.0.0
-----------------------
Parameters:
  AuthenticationProviders=wia,federated
  Config=.npmrc
  Help=False
  NonInteractive=False
  TargetConfig=
  ExpirationMinutes=129600
  ReadOnly=False
  Force=False
  Verbosity=Detailed
INI (resolved): //ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/:username=VssSessionToken
INI (resolved): //ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/:_password=<redacted ---sekretz--->
INI (resolved): //ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/:email=not-used@example.com
INI (resolved): //pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:username=VssSessionToken
INI (resolved): //pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:_password=<redacted ---sekretz--->
INI (resolved): //pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/:email=not-used@example.com
INI (resolved): # @ACCOUNT_NAME:registry=https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/
INI (resolved): # @ACCOUNT_NAME:always-auth=true
INI (resolved): @ACCOUNT_NAME:registry=https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/
INI (resolved): registry=https://registry.npmjs.org
Probing https://registry.npmjs.org/
Probe response code: 200 OK
Credential type: None.
Probing https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/ with existing credential
Probe response code: 200 OK
Credential type: Basic.
Has valid credentials: True.
Already have credentials for https://ACCOUNT_NAME.pkgs.visualstudio.com/_packaging/***-***-***-***/npm/registry/.
Probing https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/ with existing credential
Probe response code: 200 OK
Credential type: Basic.
Has valid credentials: True.
Already have credentials for https://pkgs.dev.azure.com/ACCOUNT_NAME/_packaging/***-***-***-***/npm/registry/.
leepowelldev commented 3 years ago

Yes, I believe the trick here for you was enabling the 'Switch existing organizations to use the new domain name URL' - effectively using a single url through Azure which the generated access token is valid for and transparently redirects. Sadly we can't switch to the single url for legacy reasons right now and NPM seems to be failing when telling it which registry URL to use.

leepowelldev commented 3 years ago

If anyone from the NPM team is interested in seeing this in action, I've set up a demo Azure account with pipelines setup to show it succeeding with v6, failing with v7, and failing with v7 pointing npm to the public registry. Happy to share credentials, just email me at lee@leepowell.dev

leepowelldev commented 3 years ago

This is the raw log when attempting to use the public registry upon install:

2021-02-26T15:30:29.9663086Z ##[section]Starting: Npm
2021-02-26T15:30:29.9672256Z ==============================================================================
2021-02-26T15:30:29.9672598Z Task         : npm
2021-02-26T15:30:29.9672997Z Description  : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts.
2021-02-26T15:30:29.9673370Z Version      : 1.182.0
2021-02-26T15:30:29.9673621Z Author       : Microsoft Corporation
2021-02-26T15:30:29.9673956Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/npm
2021-02-26T15:30:29.9674339Z ==============================================================================
2021-02-26T15:30:30.4508242Z [command]/opt/hostedtoolcache/node/14.15.5/x64/bin/npm --version
2021-02-26T15:30:30.7248744Z 7.6.0
2021-02-26T15:30:31.3006549Z [command]/opt/hostedtoolcache/node/14.15.5/x64/bin/npm config list
2021-02-26T15:30:31.5651593Z ; "project" config from /home/vsts/work/1/s/.npmrc
2021-02-26T15:30:31.5655227Z 
2021-02-26T15:30:31.5656620Z registry = "https://registry.npmjs.org/" 
2021-02-26T15:30:31.5657174Z 
2021-02-26T15:30:31.5657656Z ; "env" config from environment
2021-02-26T15:30:31.5658038Z 
2021-02-26T15:30:31.5658562Z userconfig = "/home/vsts/work/1/npm/26.npmrc" 
2021-02-26T15:30:31.5658930Z 
2021-02-26T15:30:31.5659399Z ; "cli" config from command line options
2021-02-26T15:30:31.5659808Z 
2021-02-26T15:30:31.5660225Z omit = [] 
2021-02-26T15:30:31.5661422Z user-agent = "npm/7.6.0 node/v14.15.5 linux x64 ci/azure-pipelines" 
2021-02-26T15:30:31.5661967Z 
2021-02-26T15:30:31.5662571Z ; node bin location = /opt/hostedtoolcache/node/14.15.5/x64/bin/node
2021-02-26T15:30:31.5663063Z ; cwd = /home/vsts/work/1/s
2021-02-26T15:30:31.5663515Z ; HOME = /home/vsts
2021-02-26T15:30:31.5664172Z ; Run `npm config ls -l` to show all defaults.
2021-02-26T15:30:31.5666093Z [command]/opt/hostedtoolcache/node/14.15.5/x64/bin/npm ci
2021-02-26T15:30:32.5071024Z npm ERR! code E401
2021-02-26T15:30:32.5072712Z npm ERR! Unable to authenticate, your authentication token seems to be invalid.
2021-02-26T15:30:32.5077247Z npm ERR! To correct this please trying logging in again with:
2021-02-26T15:30:32.5081829Z npm ERR!     npm login
2021-02-26T15:30:32.5082434Z 
2021-02-26T15:30:32.5083250Z npm ERR! A complete log of this run can be found in:
2021-02-26T15:30:32.5085094Z npm ERR!     /home/vsts/.npm/_logs/2021-02-26T15_30_32_267Z-debug.log
2021-02-26T15:30:32.5664858Z Found npm debug log, make sure the path matches with the one in npm's output: /home/vsts/.npm/_logs/2021-02-26T15_30_32_267Z-debug.log
2021-02-26T15:30:32.5665399Z 0 verbose cli [
2021-02-26T15:30:32.5665936Z 0 verbose cli   '/opt/hostedtoolcache/node/14.15.5/x64/bin/node',
2021-02-26T15:30:32.5666535Z 0 verbose cli   '/opt/hostedtoolcache/node/14.15.5/x64/bin/npm',
2021-02-26T15:30:32.5667006Z 0 verbose cli   'ci'
2021-02-26T15:30:32.5667276Z 0 verbose cli ]
2021-02-26T15:30:32.5667528Z 1 info using npm@7.6.0
2021-02-26T15:30:32.5667810Z 2 info using node@v14.15.5
2021-02-26T15:30:32.5668154Z 3 timing config:load:defaults Completed in 2ms
2021-02-26T15:30:32.5668913Z 4 timing config:load:file:/opt/hostedtoolcache/node/14.15.5/x64/lib/node_modules/npm/npmrc Completed in 1ms
2021-02-26T15:30:32.5669393Z 5 timing config:load:builtin Completed in 1ms
2021-02-26T15:30:32.5669748Z 6 timing config:load:cli Completed in 1ms
2021-02-26T15:30:32.5670108Z 7 timing config:load:env Completed in 1ms
2021-02-26T15:30:32.5670501Z 8 timing config:load:file:/home/vsts/work/1/s/.npmrc Completed in 0ms
2021-02-26T15:30:32.5670901Z 9 timing config:load:project Completed in 1ms
2021-02-26T15:30:32.5671327Z 10 timing config:load:file:/home/vsts/work/1/npm/26.npmrc Completed in 1ms
2021-02-26T15:30:32.5671733Z 11 timing config:load:user Completed in 1ms
2021-02-26T15:30:32.5672186Z 12 timing config:load:file:/opt/hostedtoolcache/node/14.15.5/x64/etc/npmrc Completed in 0ms
2021-02-26T15:30:32.5672660Z 13 timing config:load:global Completed in 0ms
2021-02-26T15:30:32.5673034Z 14 timing config:load:cafile Completed in 0ms
2021-02-26T15:30:32.5673577Z 15 timing config:load:validate Completed in 1ms
2021-02-26T15:30:32.5673950Z 16 timing config:load:setUserAgent Completed in 0ms
2021-02-26T15:30:32.5674338Z 17 timing config:load:setEnvs Completed in 1ms
2021-02-26T15:30:32.5674683Z 18 timing config:load Completed in 10ms
2021-02-26T15:30:32.5675749Z 19 verbose npm-session 3b6e854be697cfa1
2021-02-26T15:30:32.5676097Z 20 timing npm:load Completed in 16ms
2021-02-26T15:30:32.5676787Z 21 timing arborist:ctor Completed in 1ms
2021-02-26T15:30:32.5678342Z 22 timing npm-ci:rm Completed in 1ms
2021-02-26T15:30:32.5678689Z 23 timing idealTree:init Completed in 0ms
2021-02-26T15:30:32.5679063Z 24 timing idealTree:userRequests Completed in 0ms
2021-02-26T15:30:32.5679390Z 25 silly idealTree buildDeps
2021-02-26T15:30:32.5679722Z 26 timing idealTree:#root Completed in 0ms
2021-02-26T15:30:32.5680077Z 27 timing idealTree:buildDeps Completed in 1ms
2021-02-26T15:30:32.5680437Z 28 timing idealTree:fixDepFlags Completed in 0ms
2021-02-26T15:30:32.5680796Z 29 timing idealTree Completed in 3ms
2021-02-26T15:30:32.5681128Z 30 timing reify:loadTrees Completed in 6ms
2021-02-26T15:30:32.5681486Z 31 timing reify:diffTrees Completed in 1ms
2021-02-26T15:30:32.5681786Z 32 silly reify moves {}
2021-02-26T15:30:32.5682100Z 33 timing reify:retireShallow Completed in 0ms
2021-02-26T15:30:32.5682466Z 34 timing reify:createSparse Completed in 2ms
2021-02-26T15:30:32.5682815Z 35 timing reify:loadBundles Completed in 0ms
2021-02-26T15:30:32.5683691Z 36 silly tarball no local data for wrappy@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/wrappy/-/wrappy-1.0.2.tgz. Extracting by manifest.
2021-02-26T15:30:32.5684809Z 37 silly tarball no local data for rimraf@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/rimraf/-/rimraf-3.0.2.tgz. Extracting by manifest.
2021-02-26T15:30:32.5685966Z 38 silly tarball no local data for path-is-absolute@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/path-is-absolute/-/path-is-absolute-1.0.1.tgz. Extracting by manifest.
2021-02-26T15:30:32.5687072Z 39 silly tarball no local data for once@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/once/-/once-1.4.0.tgz. Extracting by manifest.
2021-02-26T15:30:32.5688152Z 40 silly tarball no local data for minimatch@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/minimatch/-/minimatch-3.0.4.tgz. Extracting by manifest.
2021-02-26T15:30:32.5689245Z 41 silly tarball no local data for inherits@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inherits/-/inherits-2.0.4.tgz. Extracting by manifest.
2021-02-26T15:30:32.5690344Z 42 silly tarball no local data for inflight@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inflight/-/inflight-1.0.6.tgz. Extracting by manifest.
2021-02-26T15:30:32.5691526Z 43 silly tarball no local data for glob@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/glob/-/glob-7.1.6.tgz. Extracting by manifest.
2021-02-26T15:30:32.5692813Z 44 silly tarball no local data for fs.realpath@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/fs.realpath/-/fs.realpath-1.0.0.tgz. Extracting by manifest.
2021-02-26T15:30:32.5693933Z 45 silly tarball no local data for concat-map@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/concat-map/-/concat-map-0.0.1.tgz. Extracting by manifest.
2021-02-26T15:30:32.5695097Z 46 silly tarball no local data for brace-expansion@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/brace-expansion/-/brace-expansion-1.1.11.tgz. Extracting by manifest.
2021-02-26T15:30:32.5696445Z 47 silly tarball no local data for balanced-match@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/balanced-match/-/balanced-match-1.0.0.tgz. Extracting by manifest.
2021-02-26T15:30:32.5697462Z 48 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/glob/-/glob-7.1.6.tgz 153ms
2021-02-26T15:30:32.5698508Z 49 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/brace-expansion/-/brace-expansion-1.1.11.tgz 138ms
2021-02-26T15:30:32.5699463Z 50 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/fs.realpath/-/fs.realpath-1.0.0.tgz 154ms
2021-02-26T15:30:32.5700354Z 51 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/rimraf/-/rimraf-3.0.2.tgz 193ms
2021-02-26T15:30:32.5701286Z 52 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/balanced-match/-/balanced-match-1.0.0.tgz 137ms
2021-02-26T15:30:32.5702207Z 53 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/concat-map/-/concat-map-0.0.1.tgz 149ms
2021-02-26T15:30:32.5703123Z 54 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inherits/-/inherits-2.0.4.tgz 179ms
2021-02-26T15:30:32.5705218Z 55 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/wrappy/-/wrappy-1.0.2.tgz 239ms
2021-02-26T15:30:32.5706127Z 56 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/once/-/once-1.4.0.tgz 227ms
2021-02-26T15:30:32.5707049Z 57 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/path-is-absolute/-/path-is-absolute-1.0.1.tgz 235ms
2021-02-26T15:30:32.5707982Z 58 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/minimatch/-/minimatch-3.0.4.tgz 232ms
2021-02-26T15:30:32.5708877Z 59 http fetch GET 401 https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inflight/-/inflight-1.0.6.tgz 254ms
2021-02-26T15:30:32.5709399Z 60 timing reify:rollback:createSparse Completed in 5ms
2021-02-26T15:30:32.5709811Z 61 timing reify:rollback:retireShallow Completed in 0ms
2021-02-26T15:30:32.5710178Z 62 timing command:ci Completed in 391ms
2021-02-26T15:30:32.5710970Z 63 verbose stack Error: Unable to authenticate, need: Basic realm="https://pkgsproduks1.pkgs.visualstudio.com/", Bearer, TFS-Federated
2021-02-26T15:30:32.5711934Z 63 verbose stack     at /opt/hostedtoolcache/node/14.15.5/x64/lib/node_modules/npm/node_modules/npm-registry-fetch/check-response.js:113:17
2021-02-26T15:30:32.5712540Z 63 verbose stack     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2021-02-26T15:30:32.5712923Z 64 verbose statusCode 401
2021-02-26T15:30:32.5713613Z 65 verbose pkgid wrappy@https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/wrappy/-/wrappy-1.0.2.tgz
2021-02-26T15:30:32.5714086Z 66 verbose cwd /home/vsts/work/1/s
2021-02-26T15:30:32.5714540Z 67 verbose Linux 5.4.0-1039-azure
2021-02-26T15:30:32.5714998Z 68 verbose argv "/opt/hostedtoolcache/node/14.15.5/x64/bin/node" "/opt/hostedtoolcache/node/14.15.5/x64/bin/npm" "ci"
2021-02-26T15:30:32.5715410Z 69 verbose node v14.15.5
2021-02-26T15:30:32.5715689Z 70 verbose npm  v7.6.0
2021-02-26T15:30:32.5715949Z 71 error code E401
2021-02-26T15:30:32.5716398Z 72 error Unable to authenticate, your authentication token seems to be invalid.
2021-02-26T15:30:32.5716859Z 73 error To correct this please trying logging in again with:
2021-02-26T15:30:32.5717212Z 73 error     npm login
2021-02-26T15:30:32.5717478Z 74 verbose exit 1
2021-02-26T15:30:32.5717602Z 
2021-02-26T15:30:32.5749897Z ##[error]Error: Npm failed with return code: 1
2021-02-26T15:30:32.5766421Z ##[section]Finishing: Npm

And this is the package-lock.json file:

{
  "name": "asdjh8923",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "balanced-match": {
      "version": "1.0.0",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/balanced-match/-/balanced-match-1.0.0.tgz",
      "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c="
    },
    "brace-expansion": {
      "version": "1.1.11",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/brace-expansion/-/brace-expansion-1.1.11.tgz",
      "integrity": "sha1-PH/L9SnYcibz0vUrlm/1Jx60Qd0=",
      "requires": {
        "balanced-match": "^1.0.0",
        "concat-map": "0.0.1"
      }
    },
    "concat-map": {
      "version": "0.0.1",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/concat-map/-/concat-map-0.0.1.tgz",
      "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
    },
    "fs.realpath": {
      "version": "1.0.0",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/fs.realpath/-/fs.realpath-1.0.0.tgz",
      "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8="
    },
    "glob": {
      "version": "7.1.6",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/glob/-/glob-7.1.6.tgz",
      "integrity": "sha1-FB8zuBp8JJLhJVlDB0gMRmeSeKY=",
      "requires": {
        "fs.realpath": "^1.0.0",
        "inflight": "^1.0.4",
        "inherits": "2",
        "minimatch": "^3.0.4",
        "once": "^1.3.0",
        "path-is-absolute": "^1.0.0"
      }
    },
    "inflight": {
      "version": "1.0.6",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inflight/-/inflight-1.0.6.tgz",
      "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
      "requires": {
        "once": "^1.3.0",
        "wrappy": "1"
      }
    },
    "inherits": {
      "version": "2.0.4",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/inherits/-/inherits-2.0.4.tgz",
      "integrity": "sha1-D6LGT5MpF8NDOg3tVTY6rjdBa3w="
    },
    "minimatch": {
      "version": "3.0.4",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/minimatch/-/minimatch-3.0.4.tgz",
      "integrity": "sha1-UWbihkV/AzBgZL5Ul+jbsMPTIIM=",
      "requires": {
        "brace-expansion": "^1.1.7"
      }
    },
    "once": {
      "version": "1.4.0",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/once/-/once-1.4.0.tgz",
      "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=",
      "requires": {
        "wrappy": "1"
      }
    },
    "path-is-absolute": {
      "version": "1.0.1",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
      "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18="
    },
    "rimraf": {
      "version": "3.0.2",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/rimraf/-/rimraf-3.0.2.tgz",
      "integrity": "sha1-8aVAK6YiCtUswSgrrBrjqkn9Bho=",
      "requires": {
        "glob": "^7.1.3"
      }
    },
    "wrappy": {
      "version": "1.0.2",
      "resolved": "https://pkgs.dev.azure.com/asdjh8923/_packaging/asdjh8923/npm/registry/wrappy/-/wrappy-1.0.2.tgz",
      "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
    }
  }
}

As you can see in the package-lock.json file, the packages were originally resolved from the Azure registry, however for this build pipeline I'm pointing the registry to npm public - it is only attempting to install rimraf (not a private package), however it is trying to authenticate against Azure and failing. I have no idea why it is trying to connect to Azure when the registry has been set to use npm public.

leepowelldev commented 3 years ago

Seems the issue also exists in Yarn: https://stackoverflow.com/questions/48258087/how-to-use-different-yarn-registry-regardless-of-registry-in-the-yarn-lock-file

We're seeing similar issues in other teams now where for local development they're using a private internal registry (which upstreams to public npm), whose URL ends up in the lock file, and then when they come to deploy via CI are unable to specify a publicly available registry to install from. They are either having to delete the lock file (less than ideal) or run a pre-build script that modifies the urls in the lock file (again not ideal).