search

raineorshine / npm-check-updates

Find newer versions of package dependencies than what your package.json allows
Other
9.33k stars 323 forks source link

ncu does not use _authToken for private, scoped packages #1330

Open regnaio opened 12 months ago

regnaio commented 12 months ago

.npmrc (for private GitLab package registry)

@<SCOPE>:registry=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/
//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken=${GITLAB_API_PERSONAL_ACCESS_TOKEN}
ncu

Checking /<PATH>/package.json
[===============-----] 3/4 75%

 @<SCOPE>/<PACKAGE> 404 Not Found - GET https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/@<SCOPE>%2f<PACKAGE> - Project not found. Either your internet connection is down or unstable and all 3 retry attempts failed, or the registry is not accessible, or the package does not exist.

The following commands succeed:

npm cache clean --force

npm i @<SCOPE>/<PACKAGE>@latest --registry=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/

Moreover, if I enter https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/@<SCOPE>%2f<PACKAGE> into my browser (logged into GitLab), it shows a good JSON with details about the package

So, it looks like ncu -u is not using _authToken in .npmrc and thus failing with a 404 when trying to find the package

I've read other related issues and PRs, but none of them resolve my issue:

raineorshine commented 12 months ago

Hi, thanks for reporting. Can you provide some additional pieces of information?

npm-check-updates reads, parses, and passes the npm config through to pacote which uses npm-registry-fetch to request versions from the registry.

Documentation: (for future reference)

regnaio commented 12 months ago

Sure!

node -v
v18.17.1
npm -v
10.1.0
ncu --verbose

╭─────────────────────────────────────────────────────────────────────────╮
│                                                                         │
│                   Update available 16.13.3 → 16.14.2                    │
│                Run npm i -g npm-check-updates to update                 │
│   https://github.com/raineorshine/npm-check-updates/compare/v16.13.3.   │
│                               ..v16.14.2                                │
│                                                                         │
╰─────────────────────────────────────────────────────────────────────────╯
Initializing
Running in local mode
Finding package file data
Checking /<PATH>/package.json

Options:
{
  args: [],
  cacheExpiration: 10,
  cacheFile: '~/.ncu-cache.json',
  cacher: undefined,
  cli: true,
  concurrency: 8,
  deep: false,
  dep: [ 'prod', 'dev', 'optional' ],
  errorLevel: 1,
  filter: undefined,
  filterVersion: undefined,
  format: [],
  install: 'prompt',
  json: false,
  loglevel: 'verbose',
  minimal: false,
  packageManager: 'npm',
  prefix: undefined,
  registryType: 'npm',
  reject: undefined,
  rejectVersion: undefined,
  retry: 3,
  target: 'latest',
  verbose: true,
  workspace: []
}

Current versions:
{
  '@<SCOPE>/node': '^0.0.13',
  '@types/node': '^20.6.0',
  'npm-run-all': '^4.1.5',
  typescript: '^5.2.2'
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

npm config (local):
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc'
}

Using npm config:
{
  registry: 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '@<SCOPE>:registry': 'https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/',
  '//gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/:_authToken': '<AUTH_TOKEN>',
  configNames: [ 'npmrc', '.npmrc' ],
  envPrefix: /^npm_config_/i,
  cwd: '/<PATH>',
  globalconfig: '/usr/local/etc/npmrc',
  userconfig: '/home/<USER>/.npmrc',
  fullMetadata: false
}

Latest versions:
{
  '@types/node': '20.6.0',
  'npm-run-all': '4.1.5',
  typescript: '5.2.2'
}

Upgraded versions:
{}

 @<SCOPE>/node  404 Not Found - GET https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/@<SCOPE>%2f<PACKAGE> - Project not found. Either your internet connection is down or unstable and all 3 retry attempts failed, or the registry is not accessible, or the package does not exist.
regnaio commented 12 months ago 
npx pacote packument @<SCOPE>/<PACKAGE>

Need to install the following packages:
pacote@17.0.4
Ok to proceed? (y) y
http fetch GET 404 https://registry.npmjs.org/@<SCOPE>%2f<PACKAGE> 179ms (cache skip)
http fetch GET 404 https://registry.npmjs.org/@<SCOPE>%2f<PACKAGE> 152ms (cache skip)
HttpErrorGeneral: 404 Not Found - GET https://registry.npmjs.org/@<SCOPE>%2f<PACKAGE> - Not found
    at /home/<USER>/.npm/_npx/d68c951c7177ed18/node_modules/npm-registry-fetch/lib/check-response.js:95:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RegistryFetcher.packument (/home/<USER>/.npm/_npx/d68c951c7177ed18/node_modules/pacote/lib/registry.js:87:19) {
  headers: [Object: null prototype] {
    date: [ 'Thu, 14 Sep 2023 16:57:13 GMT' ],
    'content-type': [ 'application/json' ],
    'content-length': [ '21' ],
    connection: [ 'keep-alive' ],
    vary: [ 'Accept-Encoding' ],
    server: [ 'cloudflare' ],
    'cf-ray': [ '806a2af57a5a15aa-SJC' ],
    'x-fetch-attempts': [ '1' ],
    'x-local-cache-status': [ 'skip' ]
  },
  statusCode: 404,
  code: 'E404',
  method: 'GET',
  uri: 'https://registry.npmjs.org/@<SCOPE>%2f<PACKAGE>',
  body: { error: 'Not found' },
  pkgid: '@<SCOPE>/<PACKAGE>@*'
}

Why is it using registry.npmjs.org? ncu shows the correct URL below gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/:

ncu

Checking /<PATH>/package.json
[===============-----] 3/4 75%

 @<SCOPE>/<PACKAGE> 404 Not Found - GET https://gitlab.com/api/v4/projects/<GITLAB_PROJECT_ID>/packages/npm/@<SCOPE>%2f<PACKAGE> - Project not found. Either your internet connection is down or unstable and all 3 retry attempts failed, or the registry is not accessible, or the package does not exist.
raineorshine commented 11 months ago

Thanks. I'll look into it. Just need to recreate the scenario.

I had assumed pacote would pick up the registry, but I guess that's not the case. So we still don't know if the issue is in npm-check-updates or pacote.

raineorshine commented 11 months ago

Can you confirm that the error only occurs for scoped packages? i.e. Do un-scoped packages on the same private registry work fine?

Also, please try npx pacote --registry=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/ packument @<SCOPE>/<PACKAGE>

regnaio commented 11 months ago 
npx pacote --registry=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/ packument @<SCOPE>/<PACKAGE>

Need to install the following packages:
pacote@17.0.4
Ok to proceed? (y) y
http fetch GET 404 https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/@<SCOPE>%2f<PACKAGE> 279ms (cache skip)
http fetch GET 404 https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/@<SCOPE>%2f<PACKAGE> 148ms (cache skip)
HttpErrorGeneral: 404 Not Found - GET https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/@<SCOPE>%2f<PACKAGE> - Project not found
    at /home/<USER>/.npm/_npx/d68c951c7177ed18/node_modules/npm-registry-fetch/lib/check-response.js:95:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RegistryFetcher.packument (/home/<USER>/.npm/_npx/d68c951c7177ed18/node_modules/pacote/lib/registry.js:87:19) {
  headers: [Object: null prototype] {
    date: [ 'Wed, 20 Sep 2023 05:51:36 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'keep-alive' ],
    'cache-control': [ 'no-cache' ],
    'content-security-policy': [ "default-src 'none'" ],
    vary: [ 'Origin, Accept-Encoding' ],
    'x-content-type-options': [ 'nosniff' ],
    'x-frame-options': [ 'SAMEORIGIN' ],
    'x-gitlab-meta': [
      '{"correlation_id":"5cc8cfef7cf4377adf43ff4527ce9e05","version":"1"}'
    ],
    'x-request-id': [ '5cc8cfef7cf4377adf43ff4527ce9e05' ],
    'x-runtime': [ '0.038748' ],
    'strict-transport-security': [ 'max-age=31536000' ],
    'referrer-policy': [ 'strict-origin-when-cross-origin' ],
    'ratelimit-observed': [ '2' ],
    'ratelimit-remaining': [ '1998' ],
    'ratelimit-reset': [ '1695189156' ],
    'ratelimit-resettime': [ 'Wed, 20 Sep 2023 05:52:36 GMT' ],
    'ratelimit-limit': [ '2000' ],
    'gitlab-lb': [ 'haproxy-main-14-lb-gprd' ],
    'gitlab-sv': [ 'localhost' ],
    'cf-cache-status': [ 'MISS' ],
    'report-to': [
      '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=oU4qawgHWzBiRMA%2FNhI4w5qWq11XXu%2Fu2n8rJDjJNjN3CK8l5SCMs4ipcK%2BidI5eR89CjEZ3Zz0BKS5EmAW1e%2FWqP7UGYls8pTuj10andzci5BgVyeEpUeCPoUg%3D"}],"group":"cf-nel","max_age":604800}'
    ],
    nel: [
      '{"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}'
    ],
    'set-cookie': [
      '_cfuvid=O35JVjVgqm19rAhx6uXdQ5Yl_.FBmO4scNUfbF6iDZE-1695189096955-0-604800000; path=/; domain=.gitlab.com; HttpOnly; Secure; SameSite=None'
    ],
    server: [ 'cloudflare' ],
    'cf-ray': [ '8097cc2f2ca01566-SJC' ],
    'content-encoding': [ 'gzip' ],
    'x-fetch-attempts': [ '1' ],
    'x-local-cache-status': [ 'skip' ]
  },
  statusCode: 404,
  code: 'E404',
  method: 'GET',
  uri: 'https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/packages/npm/@<SCOPE>%2f<PACKAGE>',
  body: { message: '404 Project Not Found', error: 'Project not found' },
  pkgid: '@<SCOPE>/<PACKAGE>@*'
}

To avoid confusion: Although I'm using ${...} variables in the URLs in the log output, the actual values were present in the log output. I just censored them out

Let me look into a non-scoped package

raineorshine commented 11 months ago

Thanks!

I'm not sure what I'm missing with the pacote call. If we could get that to work, it might clue us in to what is wrong with the npm-check-updates code.

ZenwalkerD commented 7 months ago

I am as well facing the same issue on Node 19, 21 as well especially for the scoped registry. I am able to get the packages installed via NPM from scoped registry; but NCU fails to run!

I get the error as shown:

$ ncu
Checking D:\Projects\Src\Current-Working\APP-SoftwareManager\src\Cloud\package.json
[======--------------] 17/57 29%HttpErrorGeneral: 401 Unauthorized - GET https://private-repo/api/v4/packages/npm/@simpl%2felement-icons
    at D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\node_modules\npm-registry-fetch\lib\check-response.js:95:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RegistryFetcher.packument (D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\node_modules\pacote\lib\registry.js:87:19)
    at async viewMany (D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\build\src\package-managers\npm.js:400:18)
    at async viewOne (D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\build\src\package-managers\npm.js:448:20)
    at async distTag (D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\build\src\package-managers\npm.js:586:24)
    at async getPackageVersionProtected (D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\build\src\lib\queryVersions.js:70:29)
    at async D:\Tools\NVM\v19.9.0\node_modules\npm-check-updates\node_modules\p-map\index.js:57:22 {
  headers: [Object: null prototype] {
    server: [ 'nginx' ],
    date: [ 'Mon, 22 Jan 2024 10:31:09 GMT' ],
    'content-type': [ 'application/json' ],
    'content-length': [ '30' ],
    connection: [ 'keep-alive' ],
    'cache-control': [ 'no-cache' ],
    'content-security-policy': [ "default-src 'none'" ],
    vary: [ 'Origin' ],
    'x-content-type-options': [ 'nosniff' ],
    'x-frame-options': [ 'SAMEORIGIN' ],
    'x-gitlab-meta': [ '{"correlation_id":"01HMRBJN9VRTP91QJE7GG0V99M","version":"1"}' ],
    'x-request-id': [ '01HMRBJN9VRTP91QJE7GG0V99M' ],
    'x-runtime': [ '0.214866' ],
    'x-fetch-attempts': [ '1' ]
  },
  statusCode: 401,
  code: 'E401',
  method: 'GET',
  uri: 'https://private-repo/api/v4/packages/npm/@simpl%2felement-icons',
  body: { message: '401 Unauthorized' },
  pkgid: '@simpl/element-icons@*'
}
raineorshine commented 7 months ago

Thanks, @ZenwalkerD. Sorry you're running into this issue as well.

The absolute best way to help out is to build npm-check-updates from source and do some live debugging. Unfortunately there is little I can do without being able to reproduce on my end, but I'm happy to point out the relevant code.

npm-check-updates is supposed to read in your npm config, normalize your registry settings and auth tokens, and pass them to pacote to fetch the package information. If there is a bug in this code it would be great to get to the bottom of it.