[BUG] npm@10 refuses to install packages from GitHub package registry

grahamb commented 11 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

This issue exists in the latest npm version

[X] I am using the latest npm

Current Behavior

npm@10 is not installing packages from a private GitHub Enterprise Server registry. It fails with a 403 forbidden error when trying to fetch the package. Previous versions of npm do not encounter this issue.

Expected Behavior

I should be able to install packages from a private registry when my .npmrc is configured properly.

Steps To Reproduce

Remove ~/.npm to ensure we're not pulling from cache

rm -rf ~/.npm

Configure .npmrc to use a private registry (in this case, I am using GitHub Enterprise Server 3.9.3)

//npm.github.sfu.ca/:_authToken=${GITHUB_NPM_TOKEN}
@its:registry=https://npm.github.sfu.ca/

Install a package from the private registry:

$ npm i @its/grouper-node       
npm ERR! code E403
npm ERR! 403 403 Forbidden - GET https://xxxxxxx.s3.ca-central-1.amazonaws.com/1631/npm_***?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxxxxxx&X-Amz-Date=20231125T002943Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=xxxxxxx
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy, or
npm ERR! 403 on a server you do not have access to.

npm ERR! A complete log of this run can be found in:  A complete log of this run can be found in: /home/grahamb/.npm/_logs/2023-11-25T00_29_42_285Z-debug-0.log

Debug log: 2023-11-25T00_29_42_285Z-debug-0.log

When using npm v9, this error does not occur. For example, with Node 18/npm 9:

$ node --version && npm --version
v18.17.1
9.6.7
$ rm -rf node_modules ~/.npm
$ npm i @its/grouper-node        

added 9 packages in 1s

3 packages are looking for funding
  run `npm fund` for details
npm notice 
npm notice New major version of npm available! 9.6.7 -> 10.2.4
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.2.4
npm notice Run npm install -g npm@10.2.4 to update!
npm notice

Using npm v10 with node 18 fails with the same 403.

Use npm v9 with node 21 works.

$ rm -rf node_modules ~/.npm
$ node --version && npm --version
v21.2.0
9.9.2
$ npm i @its/grouper-node        

added 9 packages in 2s

3 packages are looking for funding
  run `npm fund` for details
npm notice 
npm notice New major version of npm available! 9.9.2 -> 10.2.4
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.2.4
npm notice Run npm install -g npm@10.2.4 to update!
npm notice

This seems to indicate that something has changed with npm 10 with how it handles private registries (or at least with GitHub's registry).

Environment

npm: 10.2.4
Node.js: 20, 21
OS Name: Linux (RHEL 8)

npm config:


; "user" config from /home/grahamb/.npmrc

@grahamb:registry = "https://npm.github.sfu.ca/" @its:registry = "https://npm.github.sfu.ca/" //npm.github.sfu.ca/:_authToken = (protected)

; node bin location = /home/grahamb/.nvm/versions/node/v21.2.0/bin/node ; node version = v21.2.0 ; npm local prefix = /tmp/test ; npm version = 10.2.4 ; cwd = /tmp/test ; HOME = /home/grahamb ; Run npm config ls -l to show all defaults.

marshalys commented 11 months ago

I have the same problem, npm v9 is OK, but npm v10 got error.

bacterik commented 11 months ago

Ive also got issues with Node 18.19 and npm 10.

npm ci shell: /usr/bin/bash -e {0} env: NPM_TOKEN: GITHUB_PACKAGES_TOKEN: HTTP_PROXY: HTTPS_PROXY: npm ERR! code E400 npm ERR! 400 Bad request - GET https://npm.pkg.github.com/download/@xxx

...

Error: Process completed with exit code 1.

madmanxxx commented 11 months ago

OK, this took HOURS to debug. Environment:

Docker containers downloaded today (all are affected, because all have npm@10): node:18-buster, node:20-buster, node:20-bullseye

package-lock.json (interesting part):

"node_modules/node-csv": {
  "version": "0.5.0",
  "resolved": "https://github.com/voodootikigod/node-csv/tarball/master",
  "integrity": "sha512-dzSBhb1hyo0vijIe4ST7G0RFzqgjuIsTMDnw80+ygidl0q8ca+wBpnZH3FFWjqWHwzNJD2N7uJgfdmXks46G/A==",
  "engines": {
    "node": ">= 0.8"
  }
},

I use corporate proxy to download dependencies. 1) All other dependencies (downloaded from registry.npmjs.org) are OK, but this specific one causes cryptic HTTP 400, much like @bacterik describes. 2) If I use a direct connection (without proxy) the dependency downloads correctly. 3) Proxy does not modify the request in any way, because HTTPS.

After tricking npm to download this module from another server under my control I noticed that HTTP request line looks odd:

GET http://mytestdomain.dev:443/voodootikigod/node-csv/tarball/master HTTP/1.1

and in fact when I send a request to github.com including this atrocious line, it responds with HTTP 400. For clarity, that request should be

GET /voodootikigod/node-csv/tarball/master HTTP/1.1
Host: mytestdomain.dev

I can also confirm that downgrading npm to v9 (npm i -g npm@9) resolves this issue.

bacterik commented 11 months ago

Nice @madmanxxx. I didnt have the time myself to do the extensive work you did. But yes, I can confirm this also. I'm using a corporate proxy, and trying to fetch a package from https://npm.pkg.github.com. We have to specifiy 18.18.2 Node version from docker hub, to make it work.

lukasmahr commented 11 months ago

I have the same problem, but it does not seem to be bound to GitHub package registries. I am using Azure DevOps artifact streams and get a 403 error when using npm 10 in combination with an HTTP proxy.

Downgrading to npm 9 or removing the proxy "solves" the issue.

Edit: I could narrow it down to version 10.2.x. The previous version 10.1.0 is not affected by this problem.

madmanxxx commented 11 months ago

@lukasmahr The problem is not bound to any particular server, because the problem is within NPM. In fact I would even say that some servers (namely https://registry.npmjs.org/) are broken, because they tolerate malformed requests in the first place. By doing so the problem was very hard to debug.

Reading your post again though I have second thoughts, because the problem reported by the server is HTTP 403, which in normal circumstances means access denied. To confirm the issue is caused by NPM you may try to send a request with malformed path and see what happens.

lukasmahr commented 11 months ago

@lukasmahr The problem is not bound to any particular server, because the problem is within NPM. In fact I would even say that some servers (namely https://registry.npmjs.org/) are broken, because they tolerate malformed requests in the first place. By doing so the problem was very hard to debug.

Reading your post again though I have second thoughts, because the problem reported by the server is HTTP 403, which in normal circumstances means access denied. To confirm the issue is caused by NPM you may try to send a request with malformed path and see what happens.

@madmanxxx Indeed, it seems to be caused by the malformed path. I tested by running NPM with --verbose flag to get the URLs that return a 403. Copying and requesting exactly those URLs with curl works perfectly fine and returns a 200. So apparently at least the URL that NPM logs are okay. However, when modifying the URL by adding a character somewhere to the query params causes the 403.

shaunxu commented 10 months ago

I have exactly same issue when install packages from github private packages via proxy. I have npm 10.2.5 installde.

bodograumann commented 10 months ago

Also seeing the 400 Bad Request behind proxy with github package registry after updating npm. Did anybody find a workaround, or do I need to downgrade?

Btw: nodejs 20.9.0 seems to be the last version that bundles the working npm 10.1.0 according to https://nodejs.org/dist/index.json

raihle commented 9 months ago

I am seeing similar behavior (404 Not Found) using npm 10.2+ with private NPM registries (JFrog Artifactory) and a corporate proxy. The issue does not appear with npm 9 or 10.1, when bypassing the proxy, or when using the "normal" NPM registry - all three factors are necessary.

Request parameters in npm 10.1.0 (results in 304 Not Modified):

> GET /artifactory/api/npm/xxx-virtual/react HTTP/1.1 > accept: application/json > authorization: Basic xxxx > npm-auth-type: web > npm-command: view > pacote-pkg-id: registry:react > pacote-req-type: packument > pacote-version: 17.0.4 > user-agent: npm/10.1.0 node/v20.9.0 darwin arm64 workspaces/false > if-none-match: 66ce436a0d8e65ca1eb557c710e6c355e6543ec3 > connection: keep-alive > Accept-Encoding: gzip,deflate > Host: jfrog.xxx.io > content-length: 0

Request parameters in npm 10.2.3 (results in 404 Not Found):

> GET https://jfrog.xxx.io:443/artifactory/api/npm/xxx-virtual/react HTTP/1.1 > accept: application/json > authorization: Basic xxxx > npm-auth-type: web > npm-command: view > pacote-pkg-id: registry:react > pacote-req-type: packument > pacote-version: 17.0.4 > user-agent: npm/10.2.3 node/v20.10.0 darwin arm64 workspaces/false > if-none-match: 66ce436a0d8e65ca1eb557c710e6c355e6543ec3 > connection: keep-alive > Accept-Encoding: gzip,deflate > Host: jfrog.xxx.io > Proxy-Connection: Keep-Alive > content-length: 0

The differences are:

The inclusion of scheme, host, and port in what would normally be the path
Updated "user-agent" header
Addition of "Proxy-Connection: Keep-Alive" (this may not be due to npm version, it just seems to sometimes be there and sometimes not...)

Of these, only the first seems like a potential explanation, though it may be a red herring.

LaughDonor commented 9 months ago

I've also run into this issue with a private repository (using //repo.url/:_auth instead of :_authToken), and setting Node to the latest LTS (20.11.0) as of now, and forcing npm to use 10.1.0 (since it's bundled with 10.2.4, and 10.3.0 is the latest and still doesn't work) works for us.

hax commented 9 months ago

It seems https://github.com/npm/cli/issues/6901 is the same bug.

My current workaround:

Using node v20.0.0 (which use npm 9.x)
Using cnpm client or yarn to publish/install private packages
Do not use proxy 😜

candrews commented 9 months ago

npm (I'm using version 10.2.3) is also using the wrong protocol to make the request.

When using a proxy with the http protocol, the GET request for the .tgz file is always over the http protocol, even went package-lock.json or npm config set registry specifies an https protocol registry.

It appears that npm may be incorrectly applying the proxy's protocol to the registry URL.

gitnubster commented 9 months ago

Same issue. I have a package-lock.json with lockfile version set to 3. Giving the same headaches with npm 10.2.3 as others here, but with azure devops packages. Changing the lockfile version to 1 does something different though... First it errors with 403, but a second npm ci it succeeds... with the packages all installed ❓ ❓ ❓ Magic happens here.

Adding another package never downloaded before on the machine, and having package-lock.json set back to lockfileVersion 3 still works (but then no cache cleared or node_modules removed).

gitnubster commented 9 months ago

Behind a proxy, refreshed token, npm cache clear -force, removed project's node_modules folder, tried npm ci twice and then it seems to have packages downloaded. Subsequent installing random packages does seem to work also.

So without changing lockfileVersion. Thing I can think of is some internal variable or lower level http caching is playing a role here. Perhaps something with npm cli agent, as suggested by @rosen-dimitrov here?

grahamb commented 9 months ago

This isn't fixed yet; there's no shipped version of npm that resolves this issue. Please reopen.

ljharb commented 9 months ago

@grahamb issues on github are generally closed when the fix is merged, not when it's released.

grahamb commented 9 months ago

Yup, I'm aware of how auto-closing issues work. As the opener of this issue, however, I haven't confirmed it's fixed the issue, so it isn't "fixed" yet.

johnjardinemd commented 8 months ago

I spent a few days looking into this issue and here are my findings:

Using npm@10.5.0 I received 403s against my AzureDevops private NPM feed:

npm ERR! 403 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Using a MITM I was able to see that the following requests take place:

GET https://myorg.pkgs.visualstudio.com/_packaging/MyRepo/npm/registry/somepackage/-/somepackage-8.12.0.tgz Results in a 303 redirect to https://zrtvsblobprodeus2154.blob.core.windows.net/b-x.blob[...]

Which causes in a second GET request to the blob. The blob request includes an Authorization header and results in a 403 error:

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>AuthenticationFailed</Code>
    <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:0000000-000-0000-0000-000000000000
Time:2024-03-01T16:01:47.1290272Z</Message>
</Error>

Note that is above error was seen by our developers who authenticated using vsts-auth where a "Basic" auth token was passed as the auth header. Our build agent seems to pass a "Bearer" token and resulted in a slightly different error (this time it's a 400):

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>InvalidAuthenticationInfo</Code>
    <Message>Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:0000000-000-0000-0000-000000000000
Time:2024-03-01T16:14:56.3864768Z</Message>
</Error>

However, using npm@9.8.1 the same request to the blob does not include an Authorization header and it succeeds with a 200 response.

I even replayed my request that resulted in the 403 in postman, disabled just the Authorization header, and it succeeded.

My team has seen very intermittent behaviour. Sometimes things work, sometimes they don't. But I guess we'll stop asking questions and simply use npm@9.x until things stabilize.

heath-freenome commented 8 months ago

I gave up on this issue getting fixed and switched over all of my repos and CI/CD system to use pnpm.

raihle commented 7 months ago

This fix was included in v10.5.0 (bundled with Node 21.7.1).

gitnubster commented 7 months ago

This fix was included in v10.5.0 (bundled with Node 21.7.1).

Thanks for mentioning! I hope the fix will land very soon in node v20 and v18 as well.

grahamb commented 7 months ago

I can confirm that npm@10.5.0 fixes the issue for me as well. I second the hope for it landing in Node LTS as well.

heath-freenome commented 7 months ago

I third that 10.5.0 lands in Node 20

engineerakki commented 7 months ago

Works well with npm 10.5.0

heath-freenome commented 6 months ago

In my Dockerfiles running in a CI/CD system on GitLab I'm still seeing intermittent timeout errors:

#15 470.1 npm ERR! code ETIMEDOUT
#15 470.1 npm ERR! syscall connect
#15 470.1 npm ERR! errno ETIMEDOUT
#15 470.1 npm ERR! network request to https://registry.npmjs.org/xtend/-/xtend-2.1.2.tgz failed, reason: connect ETIMEDOUT 104.16.2.35:443
#15 470.1 npm ERR! network This is a problem related to network connectivity.
#15 470.1 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#15 470.1 npm ERR! network 
#15 470.1 npm ERR! network If you are behind a proxy, please make sure that the
#15 470.1 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#15 470.1 
#15 470.1 npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-04-05T09_00_13_641Z-debug-0.log
#15 ERROR: process "/bin/sh -c npm run artifactregistry-login\nnpm ci\n" did not complete successfully: exit code: 1

npm / cli