[BUG] `npm ci` or `npm install` hangs on my project on npm `8.11.0`

Ldoppea commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

This issue exists in the latest npm version

[X] I am using the latest npm

Current Behavior

Hi,

Running npm ci or npm install on my project hangs when using node v16.16.0 (npm v8.11.0) or later

Here is the output from my CI

Here is the URL to my project: https://github.com/cozy/cozy-keys-browser

Here is what I tested on my computer:

On node v16.15.0 (npm v8.5.5): the npm ci command success after ~50s
On node v16.16.0 (npm v8.11.0): the npm ci hangs and fails after a few minutes with Allocation failed - JavaScript heap out of memory error
- I tried to increase memory with NODE_OPTIONS="--max-old-space-size=8192" but then it just hangs way longer (more than 10min)
On node v19.0.0 (npm v8.19.2): the npm ci still hangs

I did use default NPM versions for each node version. So I did not try NPM versions between 8.5.5 and 8.11.0. So I'm not sure when the bug started.

When in this state, the terminal just display this and don't change anymore until the memory error

This bug seems to be specific to this project as my other project works perfectly with 16.16.0. So this seems to be related to my configuration.

Expected Behavior

npm ci should success after ~1min

Steps To Reproduce

In macOS Monterey (12.5.1) with nvm use 16.16.0
With this repository: https://github.com/cozy/cozy-keys-browser/
Run npm ci
The terminal should hangs

Environment

npm: 8.11.0
Node.js: 16.16.0
OS Name: MacOS Monterey 12.5.1
System Model Name: MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports)

npm config:


; "user" config from /Users/ldoppea/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /Users/ldoppea/.nvm/versions/node/v16.16.0/bin/node ; node version = v16.16.0 ; npm local prefix = /Users/ldoppea/Documents/cozy/dev/github/cozy-keys-browser ; npm version = 8.11.0 ; cwd = /Users/ldoppea/Documents/cozy/dev/github/cozy-keys-browser ; HOME = /Users/ldoppea ; Run npm config ls -l to show all defaults.



The same behavior is observed on our CI that uses Ubuntu 16.04.7 LTS

Ldoppea commented 1 year ago

I made more tests.

By trying to remove the package-lock.json to make a fresh install, I just remembered that I have to use --legacy-peer-deps on npm install.

I tried adding it to the npm ci command and it works although it was not needed on previous NPM version.

Is that intended?

Is this related to https://github.com/npm/cli/issues/5113 ?

wraithgar commented 1 year ago

What happens if you temporarily remove the git dependencies?

Ldoppea commented 1 year ago

What happens if you temporarily remove the git dependencies?

Hi,

I just did the test:

removed the duo_web_sdk plugin reference from package.json
removed the package-lock.json file
run npm install --legacy-peer-deps to generate a new package-lock.json without any ref to duo_web_sdk git repo
removed node_modules
run npm ci

I got the same behavior: the execution hangs

Torxed commented 5 months ago

This happens on random packages every now and then. We're increasing log verbosity with npm config set loglevel=silly to be sure we don't miss anything. And we've (hopefully) tried all of the mentioned "fixes" and workarounds.

And we've set all possible combinations of options:

npm config ls -l
npm config set loglevel=silly
npm config set fetch-retries 5
npm config set fetch-retry-factor 10
npm config set fetch-retry-maxtimeout 1200000
npm config set fetch-retry-mintimeout 600000
npm config set fetch-timeout 1800000
npm ci
npm run build

And despite this, we get random timeouts on packages:

npm http fetch GET 200 https://registry.npmjs.org/vue-material-design-icons/-/vue-material-design-icons-5.3.0.tgz 20548ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz 609825ms attempt #2 (cache miss)

Notably 609 825 ms.

Worth mentioning we're running this in a container as well to eliminate individual machine issues that could be the cause. And the connection is a stable gigabit fiber connection. And we're not hammering builds, it's one build at a time - consistently failing. And adding a persistent cache does eventually work around the issue for obvious reasons, but it takes numerous failed build attempts to populate the cache. And the experience is quite horrible from a CI/CD and developer perspective. (and well aware that this is to combat scrapers etc, but it breaks legit usage if that's the short answer)

stages:
  - build

build-job:
  stage: build
  environment: production
  image: node:21.6-alpine3.18
  tags: 
    - docker
  script:
    - apk add --no-cache 7zip curl git tree
    - npm config ls -l
    - npm config set loglevel=silly
    - npm config set fetch-retries 5
    - npm config set fetch-retry-factor 10
    - npm config set fetch-retry-maxtimeout 1200000
    - npm config set fetch-retry-mintimeout 600000
    - npm config set fetch-timeout 1800000
    - npm ci --cache npm_cache --prefer-offline
    - npm run build

And it fails reliably 90% of the time, with a few runs passing through.

And just to demonstrate it is different packages, the next run would say:

npm http fetch GET 200 https://registry.npmjs.org/typescript/-/typescript-4.9.5.tgz 27083ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/vue-material-design-icons/-/vue-material-design-icons-5.3.0.tgz 410857ms (cache miss)

Where 410 857 ms happens on another package.

Our full npm config before setting all the custom "fixes":

; "default" config from default values

_auth = (protected) 
access = null 
all = false 
allow-same-version = false 
also = null 
audit = true 
audit-level = null 
auth-type = "web" 
before = null 
bin-links = true 
browser = null 
ca = null 
cache = "/root/.npm" 
cache-max = null 
cache-min = 0 
cafile = null 
call = "" 
cert = null 
cidr = null 
color = true 
commit-hooks = true 
cpu = null 
depth = null 
description = true 
dev = false 
diff = [] 
diff-dst-prefix = "b/" 
diff-ignore-all-space = false 
diff-name-only = false 
diff-no-prefix = false 
diff-src-prefix = "a/" 
diff-text = false 
diff-unified = 3 
dry-run = false 
editor = "vi" 
engine-strict = false 
fetch-retries = 2 
fetch-retry-factor = 10 
fetch-retry-maxtimeout = 60000 
fetch-retry-mintimeout = 10000 
fetch-timeout = 300000 
force = false 
foreground-scripts = false 
format-package-lock = true 
fund = true 
git = "git" 
git-tag-version = true 
global = false 
global-style = false 
globalconfig = "/usr/local/etc/npmrc" 
heading = "npm" 
https-proxy = null 
if-present = false 
ignore-scripts = false 
include = [] 
include-staged = false 
include-workspace-root = false 
init-author-email = "" 
init-author-name = "" 
init-author-url = "" 
init-license = "ISC" 
init-module = "/root/.npm-init.js" 
init-version = "1.0.0" 
init.author.email = "" 
init.author.name = "" 
init.author.url = "" 
init.license = "ISC" 
init.module = "/root/.npm-init.js" 
init.version = "1.0.0" 
install-links = false 
install-strategy = "hoisted" 
json = false 
key = null 
legacy-bundling = false 
legacy-peer-deps = false 
link = false 
local-address = null 
location = "user" 
lockfile-version = null 
loglevel = "notice" 
logs-dir = null 
logs-max = 10 
; long = false ; overridden by cli
maxsockets = 15 
message = "%s" 
node-options = null 
noproxy = [""] 
npm-version = "10.2.4" 
offline = false 
omit = [] 
omit-lockfile-registry-resolved = false 
only = null 
optional = null 
os = null 
otp = null 
pack-destination = "." 
package = [] 
package-lock = true 
package-lock-only = false 
parseable = false 
prefer-dedupe = false 
prefer-offline = false 
prefer-online = false 
prefix = "/usr/local" 
preid = "" 
production = null 
progress = false 
provenance = false 
provenance-file = null 
proxy = null 
read-only = false 
rebuild-bundle = true 
registry = "https://registry.npmjs.org/" 
replace-registry-host = "npmjs" 
save = true 
save-bundle = false 
save-dev = false 
save-exact = false 
save-optional = false 
save-peer = false 
save-prefix = "^" 
save-prod = false 
sbom-format = null 
sbom-type = "library" 
scope = "" 
script-shell = null 
searchexclude = "" 
searchlimit = 20 
searchopts = "" 
searchstaleness = 900 
shell = "sh" 
shrinkwrap = true 
sign-git-commit = false 
sign-git-tag = false 
strict-peer-deps = false 
strict-ssl = true 
tag = "latest" 
tag-version-prefix = "v" 
timing = false 
umask = 0 
unicode = false 
update-notifier = true 
usage = false 
user-agent = "npm/{npm-version} node/{node-version} {platform} {arch} workspaces/{workspaces} {ci}" 
userconfig = "/root/.npmrc" 
version = false 
versions = false 
viewer = "man" 
which = null 
workspace = [] 
workspaces = null 
workspaces-update = true 
yes = null 

; "cli" config from command line options

long = true

Traceback:

npm verb stack Error: Idle timeout reached for host `registry.npmjs.org:443`
npm verb stack     at TLSSocket.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/agent/lib/agents.js:153:24)
npm verb stack     at Object.onceWrapper (node:events:633:28)
npm verb stack     at TLSSocket.emit (node:events:531:35)
npm verb stack     at Socket._onTimeout (node:net:589:8)
npm verb stack     at listOnTimeout (node:internal/timers:573:17)
npm verb stack     at process.processTimers (node:internal/timers:514:7)
npm verb cwd /root/richdocuments
npm verb Linux 6.6.10-arch1-1
npm verb node v21.6.0
npm verb npm  v10.2.4
npm ERR! code EIDLETIMEOUT
npm ERR! Idle timeout reached for host `registry.npmjs.org:443`
npm verb exit 1

One project we've used to reliably cause this is https://github.com/nextcloud/richdocuments

zainsohail88 commented 5 months ago

This happens on random packages every now and then. We're increasing log verbosity with npm config set loglevel=silly to be sure we don't miss anything. And we've (hopefully) tried all of the mentioned "fixes" and workarounds.

And we've set all possible combinations of options:

npm config ls -l
npm config set loglevel=silly
npm config set fetch-retries 5
npm config set fetch-retry-factor 10
npm config set fetch-retry-maxtimeout 1200000
npm config set fetch-retry-mintimeout 600000
npm config set fetch-timeout 1800000
npm ci
npm run build

And despite this, we get random timeouts on packages:

npm http fetch GET 200 https://registry.npmjs.org/vue-material-design-icons/-/vue-material-design-icons-5.3.0.tgz 20548ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz 609825ms attempt #2 (cache miss)

Notably 609 825 ms.

Worth mentioning we're running this in a container as well to eliminate individual machine issues that could be the cause. And the connection is a stable gigabit fiber connection. And we're not hammering builds, it's one build at a time - consistently failing. And adding a persistent cache does eventually work around the issue for obvious reasons, but it takes numerous failed build attempts to populate the cache. And the experience is quite horrible from a CI/CD and developer perspective. (and well aware that this is to combat scrapers etc, but it breaks legit usage if that's the short answer)

stages:
  - build

build-job:
  stage: build
  environment: production
  image: node:21.6-alpine3.18
  tags: 
    - docker
  script:
    - apk add --no-cache 7zip curl git tree
    - npm config ls -l
    - npm config set loglevel=silly
    - npm config set fetch-retries 5
    - npm config set fetch-retry-factor 10
    - npm config set fetch-retry-maxtimeout 1200000
    - npm config set fetch-retry-mintimeout 600000
    - npm config set fetch-timeout 1800000
    - npm ci --cache npm_cache --prefer-offline
    - npm run build

And it fails reliably 90% of the time, with a few runs passing through.

And just to demonstrate it is different packages, the next run would say:

npm http fetch GET 200 https://registry.npmjs.org/typescript/-/typescript-4.9.5.tgz 27083ms (cache miss)
npm http fetch GET 200 https://registry.npmjs.org/vue-material-design-icons/-/vue-material-design-icons-5.3.0.tgz 410857ms (cache miss)

Where 410 857 ms happens on another package.

Our full npm config before setting all the custom "fixes":

; "default" config from default values

_auth = (protected) 
access = null 
all = false 
allow-same-version = false 
also = null 
audit = true 
audit-level = null 
auth-type = "web" 
before = null 
bin-links = true 
browser = null 
ca = null 
cache = "/root/.npm" 
cache-max = null 
cache-min = 0 
cafile = null 
call = "" 
cert = null 
cidr = null 
color = true 
commit-hooks = true 
cpu = null 
depth = null 
description = true 
dev = false 
diff = [] 
diff-dst-prefix = "b/" 
diff-ignore-all-space = false 
diff-name-only = false 
diff-no-prefix = false 
diff-src-prefix = "a/" 
diff-text = false 
diff-unified = 3 
dry-run = false 
editor = "vi" 
engine-strict = false 
fetch-retries = 2 
fetch-retry-factor = 10 
fetch-retry-maxtimeout = 60000 
fetch-retry-mintimeout = 10000 
fetch-timeout = 300000 
force = false 
foreground-scripts = false 
format-package-lock = true 
fund = true 
git = "git" 
git-tag-version = true 
global = false 
global-style = false 
globalconfig = "/usr/local/etc/npmrc" 
heading = "npm" 
https-proxy = null 
if-present = false 
ignore-scripts = false 
include = [] 
include-staged = false 
include-workspace-root = false 
init-author-email = "" 
init-author-name = "" 
init-author-url = "" 
init-license = "ISC" 
init-module = "/root/.npm-init.js" 
init-version = "1.0.0" 
init.author.email = "" 
init.author.name = "" 
init.author.url = "" 
init.license = "ISC" 
init.module = "/root/.npm-init.js" 
init.version = "1.0.0" 
install-links = false 
install-strategy = "hoisted" 
json = false 
key = null 
legacy-bundling = false 
legacy-peer-deps = false 
link = false 
local-address = null 
location = "user" 
lockfile-version = null 
loglevel = "notice" 
logs-dir = null 
logs-max = 10 
; long = false ; overridden by cli
maxsockets = 15 
message = "%s" 
node-options = null 
noproxy = [""] 
npm-version = "10.2.4" 
offline = false 
omit = [] 
omit-lockfile-registry-resolved = false 
only = null 
optional = null 
os = null 
otp = null 
pack-destination = "." 
package = [] 
package-lock = true 
package-lock-only = false 
parseable = false 
prefer-dedupe = false 
prefer-offline = false 
prefer-online = false 
prefix = "/usr/local" 
preid = "" 
production = null 
progress = false 
provenance = false 
provenance-file = null 
proxy = null 
read-only = false 
rebuild-bundle = true 
registry = "https://registry.npmjs.org/" 
replace-registry-host = "npmjs" 
save = true 
save-bundle = false 
save-dev = false 
save-exact = false 
save-optional = false 
save-peer = false 
save-prefix = "^" 
save-prod = false 
sbom-format = null 
sbom-type = "library" 
scope = "" 
script-shell = null 
searchexclude = "" 
searchlimit = 20 
searchopts = "" 
searchstaleness = 900 
shell = "sh" 
shrinkwrap = true 
sign-git-commit = false 
sign-git-tag = false 
strict-peer-deps = false 
strict-ssl = true 
tag = "latest" 
tag-version-prefix = "v" 
timing = false 
umask = 0 
unicode = false 
update-notifier = true 
usage = false 
user-agent = "npm/{npm-version} node/{node-version} {platform} {arch} workspaces/{workspaces} {ci}" 
userconfig = "/root/.npmrc" 
version = false 
versions = false 
viewer = "man" 
which = null 
workspace = [] 
workspaces = null 
workspaces-update = true 
yes = null 

; "cli" config from command line options

long = true

Traceback:

npm verb stack Error: Idle timeout reached for host `registry.npmjs.org:443`
npm verb stack     at TLSSocket.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/agent/lib/agents.js:153:24)
npm verb stack     at Object.onceWrapper (node:events:633:28)
npm verb stack     at TLSSocket.emit (node:events:531:35)
npm verb stack     at Socket._onTimeout (node:net:589:8)
npm verb stack     at listOnTimeout (node:internal/timers:573:17)
npm verb stack     at process.processTimers (node:internal/timers:514:7)
npm verb cwd /root/richdocuments
npm verb Linux 6.6.10-arch1-1
npm verb node v21.6.0
npm verb npm  v10.2.4
npm ERR! code EIDLETIMEOUT
npm ERR! Idle timeout reached for host `registry.npmjs.org:443`
npm verb exit 1

One project we've used to reliably cause this is https://github.com/nextcloud/richdocuments

following: as we are also facing exactly the same issue as de, the only difference is we are using AWS CodeArtifactory instead of registry.npmjs.org:443 repositories. it seems the timeout settings are not being taken into consideration.

npm / cli