[BUG] Getting constant `FetchError...reason: Socket timeout` when installing packages

[ruluva]

Current Behavior:

When trying to install packages, without a previous cache, we get constant timeouts when trying to fetch packages, both locally and in a CI environment. The package that fails is always different, so there isn't a specific package causing the issue.

Stack trace:

19505 verbose stack FetchError: request to https://.../npm/registry/typescript/-/typescript-4.1.3.tgz failed, reason: Socket timeout
19505 verbose stack     at ClientRequest.<anonymous> (C:\npm\prefix\node_modules\npm\node_modules\minipass-fetch\lib\index.js:97:14)
19505 verbose stack     at ClientRequest.emit (events.js:315:20)
19505 verbose stack     at TLSSocket.socketErrorListener (_http_client.js:469:9)
19505 verbose stack     at TLSSocket.emit (events.js:327:22)
19505 verbose stack     at emitErrorNT (internal/streams/destroy.js:106:8)
19505 verbose stack     at emitErrorCloseNT (internal/streams/destroy.js:74:3)
19505 verbose stack     at processTicksAndRejections (internal/process/task_queues.js:80:21)
19506 verbose cwd D:\a\1\s
19507 verbose Windows_NT 10.0.17763
19508 verbose argv "C:\\hostedtoolcache\\windows\\node\\14.15.4\\x64\\node.exe" "C:\\npm\\prefix\\node_modules\\npm\\bin\\npm-cli.js" "ci" "--no-audit"
19509 verbose node v14.15.4
19510 verbose npm  v7.9.0
19511 error code ERR_SOCKET_TIMEOUT
19512 error errno ERR_SOCKET_TIMEOUT
19513 error request to https://.../npm/registry/typescript/-/typescript-4.1.3.tgz failed, reason: Socket timeout
19514 verbose exit 1

Tried using the fetch-retries, fetch-retry-mintimeout, and fetch-retry-maxtimeout options, but didn't seem to have any effect.

Expected Behavior:

Install packages without timing out.

Steps To Reproduce:

Using a project with workspaces (~6425 packages) No cache available Run npm ci

Environment:

OS: Mac OS 11.2.3 or Windows 10 Node: 14.15.4 npm: 7.9.0

[nlf]

judging by the path in your issue, i'm guessing that you are using a custom registry and not the npm one?

these socket timeouts seem like an issue with either your registry itself or the network connection between the computer you're installing packages on and your registry host. if you can, you might try installing some packages using the official npm registry just to see if you continue to have issues. if something like npm install react --registry=https://registry.npmjs.org in an empty directory still shows you socket timeout errors it seems likely the problem is with your computer. if not, sounds like it's an issue with your custom registry.

let us know what you find!

[ruluva]

Yeah, this is going through a custom Visual Studio registry. And it has been an issue on most of the machines from the team as well as the build pipelines. I can try to repro it with the npmjs registry, but it seems to only occur on bigger projects, since I didn't face the issue on a smaller one during testing.

[nlf]

you might try experimenting with the --maxsockets option too, the default is 15 but it could be you need to reduce it a bit further.

[ruluva]

So reducing might help with this issue? Is this because it might overloading it?

[nlf]

yeah, that's what i'm thinking. potentially reducing the number of concurrent requests will give the registry a better chance at handling them. it could be reducing that number enough will avoid the timeouts entirely. it's certainly worth a try, anyway.

[ruluva]

Sounds good, let me play around with a couple different values, and I'll post any results.

[ruluva]

So I tried 10, 5, and 1 for maxsockets, and I was still getting the same timeout issues.

[nlf]

interesting.. so we're left with trying to reproduce this against the npm registry, which as you mentioned is somewhat difficult. installing the react package fetches quite a few dependencies and may be enough to trigger it for you.

do you have a proxy in use, by chance? i'm just trying to determine what else may be in the network between the computer doing the installation and the registry itself.

[josephliccini]

@nlf this is a MSFT internal registry, we could get a on a Teams call and debug if that's helpful?

[ruluva]

So we followed up with the Azure Devops team in terms of our registry, to get an idea of whether that was causing an issue, and from their side, they aren't even seeing the timeout calls. So they believe it might be some logic in NPM that might be throwing a timeout for the wrong reason. I tried to replicate this by setting up a Fiddler proxy to see the NPM get requests, and I see the successful ones as well as the redirects to the blob storage associated with them, but I never saw the request that timed out.

[nlf]

so we're timing out requests that were never even actually sent? that's.. surprising, and strange.

do you, by chance, use vsts-npm-auth?

[ruluva]

Yeah, I can share the fiddler trace if that helps.

Most of the team does, but there are a couple people on the team that have Macs, so we set the .npmrc authentication manually.

[nlf]

Most of the team does, but there are a couple people on the team that have Macs, so we set the .npmrc authentication manually.

hmm, ok. and you already said the folks on macs are seeing this problem too, so we definitely can't point to vsts-npm-auth as a source of the problem.

Yeah, I can share the fiddler trace if that helps.

i'll reach out to you in a higher bandwidth way either this afternoon or tomorrow morning (pacific timezone) so we can try to debug a bit more.

[ruluva]

Yeah, definitely seeing this issue as well on my Mac.

Sounds good, I'm on Teams if that helps or whichever way is fine.

[Keysox]

We're seeing this same error with these versions:

• node 15.11.0 / npm 7.6.0
• node 16.0.0 / npm 7.10.0

The interesting thing is that it'll timeout within 4 seconds of a previous 200 and not retry at all when a 502 is encountered using the defaults specified here: https://docs.npmjs.com/cli/v7/using-npm/config#fetch-retries.

Is there any update on this issue or workarounds we should try?

[darcyclarke]

@nlf / @ruluva any update here?

[nlf]

@nlf / @ruluva any update here?

not yet, we've been trying to debug async. i have a few theories but so far nothing has panned out

[nlf]

i think this is potentially related to the patch i submitted to minipass-fetch here: https://github.com/npm/minipass-fetch/pull/14

it is most definitely related to the patch i submitted to make-fetch-happen here: https://github.com/npm/make-fetch-happen/pull/54

the first of those fixes helps mostly in cb() never called situations, though issues there can certainly manifest in strange ways since it can cause a request to be neither resolved nor rejected. the second fix will allow make-fetch-happen to retry requests that are destroyed due to a socket idle timeout within agentkeepalive.

to address the socket timeouts, you can try passing --fetch-timeout=600000 on the command line, which adjusts the connection timeout as well as the idle timeout. the default value is 5 minutes, which i would assume would be log enough, but clearly that's not the case.

[mehulkar]

👋🏾 I don't know if it's the same issue, but our internal Artifactory instance is also seeing npm ERR! code ERR_SOCKET_TIMEOUT errors sporadically on random modules after upgrading to npm@7.10.0. We had never had this issue with npm@6. (I am unable to test newer versions in a real world project, because after 7.10.0, I run into #3284.)

[MikaelLambert]

Thanks @mehulkar, I ran into a similar issue related to our internal Artifactory as well, but your workaround worked for me. I was also able to workaround the issue by upgrading to the latest version of Node/npm. I'm not sure if the latest npm has increased timeouts or different fetching behaviour as described by @nlf , but just passing on the information to help out the investigation.

Here's my full results for 2962 packages:

(Using npm registry) Node.js 15.14.0 npm 7.7.6 Build time: 37.194s

(Internal Artifactory) Node.js 15.14.0 npm 7.7.6 Socket timeout 5 minutes

(Internal Artifactory) Node.js 14.17.3 npm 6.14.13 Build time: 99.694s

(Internal Artifactory) Node.js 16.5.0 npm 7.19.1 Build time: 6 minutes

[ansf]

I am seeing this too, using npm 7.24.0 with registry.npmjs.org.

with npm -dd ci the log around the error looks like this:

#10 163.1 npm http fetch GET 200 https://registry.npmjs.org/googleapis/-/googleapis-87.0.0.tgz 144293ms (cache miss)
#10 163.1 npm timing reifyNode:node_modules/googleapis Completed in 153287ms
#10 174.9 npm http fetch GET 200 https://registry.npmjs.org/date-fns/-/date-fns-2.24.0.tgz 154055ms (cache miss)
#10 174.9 npm timing reifyNode:node_modules/date-fns Completed in 165040ms
#10 189.3 npm timing reify:rollback:createSparse Completed in 14368ms
#10 189.3 npm timing reify:rollback:retireShallow Completed in 0ms
#10 189.3 npm timing command:ci Completed in 187602ms
#10 189.3 npm notice 
#10 189.3 npm notice New patch version of npm available! 7.24.0 -> 7.24.1
#10 189.3 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.24.1>
#10 189.3 npm notice Run `npm install -g npm@7.24.1` to update!
#10 189.3 npm notice 
#10 189.3 npm verb stack Error: Socket timeout
#10 189.3 npm verb stack     at TLSSocket.onTimeout (/usr/local/lib/node_modules/npm/node_modules/agentkeepalive/lib/agent.js:345:23)
#10 189.3 npm verb stack     at TLSSocket.emit (node:events:402:35)
#10 189.3 npm verb stack     at TLSSocket.Socket._onTimeout (node:net:486:8)
#10 189.3 npm verb stack     at listOnTimeout (node:internal/timers:557:17)
#10 189.3 npm verb stack     at processTimers (node:internal/timers:500:7)
#10 189.3 npm verb cwd /build
#10 189.3 npm verb Linux 5.10.47-linuxkit
#10 189.3 npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "-dd" "ci"
#10 189.3 npm verb node v16.10.0
#10 189.3 npm verb npm  v7.24.0
#10 189.3 npm ERR! code ERR_SOCKET_TIMEOUT
#10 189.3 npm ERR! network Socket timeout
#10 189.3 npm ERR! network This is a problem related to network connectivity.
#10 189.3 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#10 189.3 npm ERR! network 
#10 189.3 npm ERR! network If you are behind a proxy, please make sure that the
#10 189.3 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#10 189.3 npm verb exit 1

I find it interesting that we see in the log command:ci Completed and the timeout happens afterwards.

Let me know if I can provide any additional information to help identify the issue!

[dnorrstrand]

I am getting the same socket time-out error on a Mac:

 > [5/6] RUN npm i:                                                             
#13 23.18 npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated                                                                  
#13 24.45 npm WARN deprecated svgo@1.3.2: This SVGO version is no longer supported. Upgrade to v2.x.x.                                                          
#13 26.05 npm WARN deprecated sane@4.1.0: some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
#13 26.40 npm WARN deprecated rollup-plugin-babel@4.4.0: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-babel.
#13 27.47 npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
#13 27.74 npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
#13 29.64 npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
#13 47.39 npm WARN deprecated har-validator@5.1.5: this library is no longer supported
#13 48.63 npm WARN deprecated flatten@1.0.3: flatten is deprecated in favor of utility frameworks such as lodash.
#13 57.48 npm WARN deprecated babel-eslint@10.1.0: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
#13 63.27 npm WARN deprecated @hapi/topo@3.1.6: This version has been deprecated and is no longer supported or maintained
#13 63.28 npm WARN deprecated @hapi/hoek@8.5.1: This version has been deprecated and is no longer supported or maintained
#13 63.46 npm WARN deprecated @hapi/bourne@1.3.2: This version has been deprecated and is no longer supported or maintained
#13 63.67 npm WARN deprecated @hapi/address@2.1.4: Moved to 'npm install @sideway/address'
#13 66.17 npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
#13 66.73 npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
#13 67.23 npm WARN deprecated @hapi/joi@15.1.1: Switch to 'npm install joi'
#13 67.27 npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
#13 69.57 npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
#13 76.99 npm WARN deprecated browserslist@2.11.3: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
#13 80.30 npm WARN deprecated core-js@2.6.12: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
#13 85.96 npm notice 
#13 85.96 npm notice New minor version of npm available! 8.1.4 -> 8.2.0
#13 85.96 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.2.0>
#13 85.96 npm notice Run `npm install -g npm@8.2.0` to update!
#13 85.96 npm notice 
#13 85.96 npm ERR! code ERR_SOCKET_TIMEOUT
#13 85.96 npm ERR! network Socket timeout
#13 85.96 npm ERR! network This is a problem related to network connectivity.
#13 85.96 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#13 85.96 npm ERR! network 
#13 85.96 npm ERR! network If you are behind a proxy, please make sure that the
#13 85.96 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#13 86.03 
#13 86.03 npm ERR! A complete log of this run can be found in:
#13 86.03 npm ERR!     /root/.npm/_logs/2021-12-07T14_41_58_610Z-debug.log
------
executor failed running [/bin/sh -c npm i]: exit code: 1

Interestingly, I have just updated to the latest npm version 8.2.0:

➜  be-frontend-react git:(master) ✗ npm -v
8.2.0

yet the npm notice still recommends updating:

#13 85.96 npm notice New minor version of npm available! 8.1.4 -> 8.2.0
#13 85.96 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.2.0>
#13 85.96 npm notice Run `npm install -g npm@8.2.0` to update!

Is there a fix available yet?

[morsik]

I just encountered same problem as 2 comments above with npm 8.1.0.

Getting Socket timeout right after everything was correctly downloaded with 200 OK. No idea what npm tries to connect when it gets this timeout as error message doesn't provide such useful formation :(

[ansf]

few days ago a longstanding race condition which also caused timeouts, was fixed in agentkeepalive 4.2.0: https://github.com/node-modules/agentkeepalive/issues/57

i don't know if it is really this issue, but since it appears in the stacktrace in https://github.com/npm/cli/issues/3078#issuecomment-927572391, maybe we can try and update?

[nlf]

we shipped the latest agentkeepalive in npm@8.3.1

can any of you who were seeing this consistently give that a shot and see if it's any better?

follow up question, i'm assuming the answer is "no" due to the use of agentkeepalive in the stack traces i'm seeing here, but are any of you experiencing timeouts using a proxy?

[avpengage]

Hi, I found this thread while trying to figure out why I was getting network timeout errors with NPM v7+.

I believe I'm seeing the same issues. I can do npm install and everything works fine with npm v6.14.15.

Since then I've upgraded to npm 8.1.2 and I get timeout error during install.

I've just tried v8.4.1 and still getting timeout issues:

(I have no proxy configured. I've tried with the --force option as well. Anything else I can try to help get to the bottom of this?)

(sorry - my code block won't show as code... One of those days I guess...)

`D:\src\tmp\another_rmap2\rmap-server [main ≡ +0 ~2 -0 !]> npm i npm WARN old lockfile npm WARN old lockfile The package-lock.json file was created with an old version of npm, npm WARN old lockfile so supplemental metadata must be fetched from the registry. npm WARN old lockfile npm WARN old lockfile This is a one-time fix-up, please be patient... npm WARN old lockfile npm WARN old lockfile Error: git dep preparation failed npm WARN old lockfile at ChildProcess. (C:\Program Files\nodejs\node_modules\npm\node_modules\@npmcli\promise-spawn\index.js:64:27) npm WARN old lockfile at ChildProcess.emit (node:events:390:28) npm WARN old lockfile at maybeClose (node:internal/child_process:1064:16) npm WARN old lockfile at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5) npm WARN old lockfile Could not fetch metadata for ais-decoder-ts@git+ssh://git@github.com/EngageFCP/ais-protocol-decoding.git#be714720de784d6ce0ce309b94d18e1eca3d129b Error: git dep preparation failed npm WARN old lockfile at ChildProcess. (C:\Program Files\nodejs\node_modules\npm\node_modules\@npmcli\promise-spawn\index.js:64:27) npm WARN old lockfile at ChildProcess.emit (node:events:390:28) npm WARN old lockfile at maybeClose (node:internal/child_process:1064:16) npm WARN old lockfile at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5) { npm WARN old lockfile cmd: 'C:\Program Files\nodejs\node.exe', npm WARN old lockfile args: [ npm WARN old lockfile 'C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js', npm WARN old lockfile 'install', npm WARN old lockfile '--force', npm WARN old lockfile '--cache=C:\Users\antho\AppData\Local\npm-cache', npm WARN old lockfile '--prefer-offline=false', npm WARN old lockfile '--prefer-online=false', npm WARN old lockfile '--offline=false', npm WARN old lockfile '--no-progress', npm WARN old lockfile '--no-save', npm WARN old lockfile '--no-audit', npm WARN old lockfile '--include=dev', npm WARN old lockfile '--include=peer', npm WARN old lockfile '--include=optional', npm WARN old lockfile '--no-package-lock-only', npm WARN old lockfile '--no-dry-run' npm WARN old lockfile ], npm WARN old lockfile code: 1, npm WARN old lockfile signal: null, npm WARN old lockfile stdout: '', npm WARN old lockfile stderr: 'npm WARN using --force Recommended protections disabled.\n' + npm WARN old lockfile 'npm WARN old lockfile \n' + npm WARN old lockfile 'npm WARN old lockfile The package-lock.json file was created with an old version of npm,\n' + npm WARN old lockfile 'npm WARN old lockfile so supplemental metadata must be fetched from the registry.\n' + npm WARN old lockfile 'npm WARN old lockfile \n' + npm WARN old lockfile 'npm WARN old lockfile This is a one-time fix-up, please be patient...\n' + npm WARN old lockfile 'npm WARN old lockfile \n' + npm WARN old lockfile 'npm ERR! code ETIMEDOUT\n' + npm WARN old lockfile 'npm ERR! syscall connect\n' + npm WARN old lockfile 'npm ERR! errno ETIMEDOUT\n' + npm WARN old lockfile 'npm ERR! network request to https://npm.saildrone.com/typescript/-/typescript-3.9.5.tgz failed, reason: connect ETIMEDOUT 54.177.189.88:443\n' + npm WARN old lockfile 'npm ERR! network This is a problem related to network connectivity.\n' + npm WARN old lockfile 'npm ERR! network In most cases you are behind a proxy or have bad network settings.\n' + npm WARN old lockfile 'npm ERR! network \n' + npm WARN old lockfile 'npm ERR! network If you are behind a proxy, please make sure that the\n' + npm WARN old lockfile "npm ERR! network 'proxy' config is set properly. See: 'npm help config'\n" + npm WARN old lockfile '\n' + npm WARN old lockfile 'npm ERR! A complete log of this run can be found in:\n' + npm WARN old lockfile 'npm ERR! C:\Users\antho\AppData\Local\npm-cache\_logs\2022-02-08T20_01_41_118Z-debug-0.log\n' npm WARN old lockfile } npm ERR! code 1 npm ERR! git dep preparation failed npm ERR! command C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js install --force --cache=C:\Users\antho\AppData\Local\npm-cache --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run npm ERR! npm WARN using --force Recommended protections disabled. npm ERR! npm WARN old lockfile npm ERR! npm WARN old lockfile The package-lock.json file was created with an old version of npm, npm ERR! npm WARN old lockfile so supplemental metadata must be fetched from the registry. npm ERR! npm WARN old lockfile npm ERR! npm WARN old lockfile This is a one-time fix-up, please be patient... npm ERR! npm WARN old lockfile npm ERR! npm ERR! code ETIMEDOUT npm ERR! npm ERR! syscall connect npm ERR! npm ERR! errno ETIMEDOUT npm ERR! npm ERR! network request to https://npm.saildrone.com/typescript/-/typescript-3.9.5.tgz failed, reason: connect ETIMEDOUT 54.177.189.88:443 npm ERR! npm ERR! network This is a problem related to network connectivity. npm ERR! npm ERR! network In most cases you are behind a proxy or have bad network settings. npm ERR! npm ERR! network npm ERR! npm ERR! network If you are behind a proxy, please make sure that the npm ERR! npm ERR! network 'proxy' config is set properly. See: 'npm help config' npm ERR! npm ERR! npm ERR! A complete log of this run can be found in: npm ERR! npm ERR! C:\Users\antho\AppData\Local\npm-cache_logs\2022-02-08T20_03_57_950Z-debug-0.log

npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\antho\AppData\Local\npm-cache_logs\2022-02-08T20_01_31_123Z-debug-0.log D:\src\tmp\another_rmap2\rmap-server [main ≡ +0 ~2 -0 ! `

[avpengage]

Here's the log from a clean run (package-lock.json deleted):

`D:\src\tmp\another_rmap2\rmap-server [main ≡ +0 ~1 -1 !]> npm i npm ERR! code 1 npm ERR! git dep preparation failed npm ERR! command C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js install --force --cache=C:\Users\antho\AppData\Local\npm-cache --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run npm ERR! npm WARN using --force Recommended protections disabled. npm ERR! npm WARN old lockfile npm ERR! npm WARN old lockfile The package-lock.json file was created with an old version of npm, npm ERR! npm WARN old lockfile so supplemental metadata must be fetched from the registry. npm ERR! npm WARN old lockfile npm ERR! npm WARN old lockfile This is a one-time fix-up, please be patient... npm ERR! npm WARN old lockfile npm ERR! npm ERR! code ETIMEDOUT npm ERR! npm ERR! syscall connect npm ERR! npm ERR! errno ETIMEDOUT npm ERR! npm ERR! network request to https://npm.saildrone.com/typescript/-/typescript-3.9.5.tgz failed, reason: connect ETIMEDOUT 54.177.189.88:443 npm ERR! npm ERR! network This is a problem related to network connectivity. npm ERR! npm ERR! network In most cases you are behind a proxy or have bad network settings. npm ERR! npm ERR! network npm ERR! npm ERR! network If you are behind a proxy, please make sure that the npm ERR! npm ERR! network 'proxy' config is set properly. See: 'npm help config' npm ERR! npm ERR! npm ERR! A complete log of this run can be found in: npm ERR! npm ERR! C:\Users\antho\AppData\Local\npm-cache_logs\2022-02-08T20_16_34_988Z-debug-0.log

npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\antho\AppData\Local\npm-cache_logs\2022-02-08T20_16_30_863Z-debug-0.log D:\src\tmp\another_rmap2\rmap-server [main ≡ +0 ~1 -1 !]>`

[CodeTroopers]

@nlf

follow up question, i'm assuming the answer is "no" due to the use of agentkeepalive in the stack traces i'm seeing here, but are any of you experiencing timeouts using a proxy?

I have the error with NPM > 8.3.1 I open a new issue #4414

[the-vampiire]

i am getting this issue without using a custom registry. all regular NPM hosted packages.

no idea how to fix it and its frustrating. i have to run the build 10 times and hope one goes through.

tried all sorts of tricks:

• running the global update of npm
• adding custom config settings (just increased everything i could)

RUN npm config set \
    fetch-retry-maxtimeout=120000 \
    fetch-retry-mintimeout=10000 \
    fetch-timeout=600000 \
    fetch-retries=10

• removing proxy settings (i dont have any proxies running but just a sanity check)

RUN npm config rm proxy https-proxy

really at a loss here. this happens on both node 16 and 17 Docker base images

[hanzlamateen]

i am getting this issue without using a custom registry. all regular NPM hosted packages.

no idea how to fix it and its frustrating. i have to run the build 10 times and hope one goes through.

tried all sorts of tricks:

• running the global update of npm
• adding custom config settings (just increased everything i could)

RUN npm config set \
    fetch-retry-maxtimeout=120000 \
    fetch-retry-mintimeout=10000 \
    fetch-timeout=600000 \
    fetch-retries=10

• removing proxy settings (i dont have any proxies running but just a sanity check)

RUN npm config rm proxy https-proxy

really at a loss here. this happens on both node 16 and 17 Docker base images

Getting same issue

[cons0l3]

node 16.13.2, npm 8.4.1

when installing @aws-amplify/cli I inconsistently run into ERR_SOCKET_TIMEOUT as well. Tempering with npm config for fetch-retry-{max,min}timeout et al. did not help. removing a non existing procy config did also not help.

[ohammami1]

a rather annoying issue, this is actually crashing my whole CI/CD Pipeline while I'm pretty sure I don't have much of a connectivity problem

Step 6/15 : RUN npm install -g @angular/cli     &&     npm install --save --force
 ---> Running in 8ccd1b51c75d
npm notice 
npm notice New minor version of npm available! 8.3.1 -> 8.5.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.0>
npm notice Run `npm install -g npm@8.5.0` to update!
npm notice 
npm ERR! code ERR_SOCKET_TIMEOUT
npm ERR! errno ERR_SOCKET_TIMEOUT
npm ERR! network Invalid response body while trying to fetch https://registry.npmjs.org/hosted-git-info: Socket timeout
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-02-17T13_49_21_066Z-debug-0.log

npm is from docker node:lts, which should be the stable version

[simonheavenvolvo]

Encountered this problem today, in my case I can change to use npm ci which has worked. My condolences if that isn't suitable for what you need.

[martinburger]

@cons0l3

when installing @aws-amplify/cli I inconsistently run into ERR_SOCKET_TIMEOUT as well.

I have been trying to install that package, too. So far without any success. Has anybody found any workaround to install the Amplify CLI package?

[the-vampiire]

i have no reasonable solution. the workaround i have had to use (which is ridiculous tbh, but i cant have 90% random failed builds):

1. npm install on the host
2. COPY node_modules node_modules (in place of the RUN npm install directive)
3. continue

i am having this issue across both macOS / docker desktop and ubuntu. i clearly do not have any connection issues because i am able to install fine locally on the host. something fishy is going on inside the dockerfile build environment.

[CodeTroopers]

@the-vampiire could you confirm that it works with npm 8.3.O and fails with NPM >=8.3.1 like described in my issue #4414

[ohammami1]

i have no reasonable solution. the workaround i have had to use (which is ridiculous tbh, but i cant have 90% random failed builds):

1. `npm install` on the host

2. `COPY node_modules node_modules` (in place of the `RUN npm install` directive)

3. continue

i am having this issue across both macOS / docker desktop and ubuntu. i clearly do not have any connection issues because i am able to install fine locally on the host. something fishy is going on inside the dockerfile build environment.

that's basically the solution I implemented on my CI/CD pipeline, since .npmrc timeout option does not solve the issue I just made sure to persist the node_modules and $HOME/.npm directories as a workaround, still I'm not very sure if this solution won't break in the future. This is more of a hack since I can't mount volumes in dockerfiles, and can't call the whole build thing(ng build) from an entrypoint since it's a multistaged build so really.. I'll be watching closely for a bug fix for this one. I also tested the old node 8.0/8.5 docker images, the downloads worked just fine, so I'm pretty sure this is a regression in the newer versions of npm and not a server timeout.

[the-vampiire]

i have no reasonable solution. the workaround i have had to use (which is ridiculous tbh, but i cant have 90% random failed builds):

1. `npm install` on the host

2. `COPY node_modules node_modules` (in place of the `RUN npm install` directive)

3. continue

i am having this issue across both macOS / docker desktop and ubuntu. i clearly do not have any connection issues because i am able to install fine locally on the host. something fishy is going on inside the dockerfile build environment.

that's basically the solution I implemented on my CI/CD pipeline, since .npmrc timeout option does not solve the issue I just made sure to persist the node_modules and $HOME/.npm directories as a workaround, still I'm not very sure if this solution won't break in the future. This is more of a hack since I can't mount volumes in dockerfiles, and can't call the whole build thing(ng build) from an entrypoint since it's a multistaged build so really.. I'll be watching closely for a bug fix for this one. I also tested the old node 8.0/8.5 docker images, the downloads worked just fine, so I'm pretty sure this is a regression in the newer versions of npm and not a server timeout.

every where else i reported this issue told me i was crazy and dismissed it. it sucks but at least reassuring im not crazy lol.

out of curiosity would yarn fix this? it reports as a network issue but clearly it isnt given that the host can get the data without issue. ive never yarn but if it resolves the issue it may be time to start.

[martinburger]

@the-vampiire could you confirm that it works with npm 8.3.O and fails with NPM >=8.3.1 like described in my issue #4414

FYI: I installed the Node packages on macOS using Homebrew.

With both Node v17.5.0 / v16.14.0 and NPM 8.4.1 / 8.3.1 I get that ERR_SOCKET_TIMEOUT.

With v14.19.0 and 6.14.16 I've been able to install @aws-amplify/cli aside from some npm WARNings:

npm WARN relay-compiler@12.0.0 requires a peer of graphql@^15.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @aws-amplify/codegen-ui-react@1.2.0 requires a peer of react@^17.0.2 but none is installed. You must install peer dependencies yourself.
npm WARN @aws-amplify/codegen-ui-react@1.2.0 requires a peer of react-dom@^17.0.2 but none is installed. You must install peer dependencies yourself.

[the-vampiire]

@martinburger @CodeTroopers confirmed still failing on 8.3.0:

regular npm install

dockerfile

FROM node:17-alpine as prod-dependencies

COPY package*.json ./

RUN npm install --only=production

# all (prod and dev dependencies)
# copies prod deps and installs dev deps
FROM node:17-alpine as all-dependencies

COPY --from=prod-dependencies ./ ./

RUN npm install

# build stage
FROM node:17-alpine as compile

WORKDIR /compile

COPY --from=all-dependencies ./ ./

# ... rest ireelevant to test of bug ...

output

 docker build -t npm-cli-bug .
[+] Building 25.3s (8/12)
 => [internal] load build definition from Dockerfile                                                                                                                          0.0s
 => => transferring dockerfile: 744B                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/node:17-alpine                                                                                                             1.7s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                   0.0s
 => [internal] load build context                                                                                                                                             0.1s
 => => transferring context: 1.06MB                                                                                                                                           0.0s
 => CACHED [prod-dependencies 1/3] FROM docker.io/library/node:17-alpine@sha256:0e83c810225bc29e614189acf3d6419e3c09881cefb9f7a170fdcfe3e15bbfd5                              0.0s
 => [prod-dependencies 2/3] COPY package*.json ./                                                                                                                             0.0s
 => ERROR [prod-dependencies 3/3] RUN npm install --only=production                                                                                                          23.4s
------
 > [prod-dependencies 3/3] RUN npm install --only=production:
#9 6.019 npm WARN deprecated formidable@1.2.6: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
#9 6.058 npm WARN tarball tarball data for nftu-schema@git+ssh://git@github.com/nftoolkit/nftu-schema.git#a233c019db73c3bfedd4afffa906dea526833e1d (null) seems to be corrupted. Trying again.
#9 6.694 npm WARN tarball tarball data for nftu-schema@git+ssh://git@github.com/nftoolkit/nftu-schema.git#a233c019db73c3bfedd4afffa906dea526833e1d (null) seems to be corrupted. Trying again.
#9 6.891 npm WARN deprecated superagent@6.1.0: Please upgrade to v7.0.2+ of superagent.  We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing.  See the releases tab for more information at <https://github.com/visionmedia/superagent/releases>. Thanks to @shadowgate15, @spence-s, and @niftylettuce. Superagent is sponsored by Forward Email at <https://forwardemail.net>.
#9 7.799 npm WARN deprecated discord-api-types@0.26.1: No longer supported. Install the latest release!
#9 23.29 npm notice
#9 23.29 npm notice New minor version of npm available! 8.4.1 -> 8.5.1
#9 23.29 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.1>
#9 23.29 npm notice Run `npm install -g npm@8.5.1` to update!
#9 23.29 npm notice
#9 23.30 npm ERR! code ENOENT
#9 23.30 npm ERR! syscall spawn git
#9 23.30 npm ERR! path git
#9 23.30 npm ERR! errno -2
#9 23.30 npm ERR! enoent An unknown git error occurred
#9 23.30 npm ERR! enoent This is related to npm not being able to find a file.
#9 23.30 npm ERR! enoent
#9 23.30
#9 23.30 npm ERR! A complete log of this run can be found in:
#9 23.30 npm ERR!     /root/.npm/_logs/2022-02-20T16_26_41_453Z-debug-0.log
------
executor failed running [/bin/sh -c npm install --only=production]: exit code: 254

downgrade to 8.3.0

dockerfile

FROM node:17-alpine as prod-dependencies

COPY package*.json ./

RUN npm install -g npm@8.3.0

RUN npm install --only=production

# all (prod and dev dependencies)
# copies prod deps and installs dev deps
FROM node:17-alpine as all-dependencies

COPY --from=prod-dependencies ./ ./

RUN npm install -g npm@8.3.0

RUN npm install

# build stage
FROM node:17-alpine as compile

WORKDIR /compile

COPY --from=all-dependencies ./ ./

output

docker build -t npm-cli-bug .
[+] Building 29.0s (8/13)
 => [internal] load build definition from Dockerfile                                                                                                                          0.0s
 => => transferring dockerfile: 473B                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/node:17-alpine                                                                                                             0.9s
 => [internal] load build context                                                                                                                                             0.0s
 => => transferring context: 72B                                                                                                                                              0.0s
 => [prod-dependencies 1/4] FROM docker.io/library/node:17-alpine@sha256:0e83c810225bc29e614189acf3d6419e3c09881cefb9f7a170fdcfe3e15bbfd5                                     0.0s
 => CACHED [prod-dependencies 2/4] COPY package*.json ./                                                                                                                      0.0s
 => [prod-dependencies 3/4] RUN npm install -g npm@8.3.0                                                                                                                      5.1s
 => ERROR [prod-dependencies 4/4] RUN npm install --only=production                                                                                                          22.9s
------
 > [prod-dependencies 4/4] RUN npm install --only=production:
#9 5.772 npm WARN tarball tarball data for nftu-schema@git+ssh://git@github.com/nftoolkit/nftu-schema.git#a233c019db73c3bfedd4afffa906dea526833e1d (null) seems to be corrupted. Trying again.
#9 5.883 npm WARN deprecated formidable@1.2.6: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
#9 6.425 npm WARN tarball tarball data for nftu-schema@git+ssh://git@github.com/nftoolkit/nftu-schema.git#a233c019db73c3bfedd4afffa906dea526833e1d (null) seems to be corrupted. Trying again.
#9 6.745 npm WARN deprecated superagent@6.1.0: Please upgrade to v7.0.2+ of superagent.  We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing.  See the releases tab for more information at <https://github.com/visionmedia/superagent/releases>. Thanks to @shadowgate15, @spence-s, and @niftylettuce. Superagent is sponsored by Forward Email at <https://forwardemail.net>.
#9 7.700 npm WARN deprecated discord-api-types@0.26.1: No longer supported. Install the latest release!
#9 22.75 npm ERR! code ENOENT
#9 22.75 npm ERR! syscall spawn git
#9 22.75 npm ERR! path git
#9 22.75 npm ERR! errno -2
#9 22.75 npm ERR! enoent An unknown git error occurred
#9 22.75 npm ERR! enoent This is related to npm not being able to find a file.
#9 22.75 npm ERR! enoent
#9 22.75
#9 22.75 npm ERR! A complete log of this run can be found in:
#9 22.75 npm ERR!     /root/.npm/_logs/2022-02-20T16_30_22_232Z-debug-0.log
------
executor failed running [/bin/sh -c npm install --only=production]: exit code: 254

[CodeTroopers]

@the-vampiire your errors are not due to socket timeout. Try to clean your cache before perform a clean install

[the-vampiire]

Oh boy I’m a clown. Tried to slap a PoC together quickly before leaving yesterday morning and grabbed the wrong package.json. Saw ERR and copied logs to leave 🤦‍♂️

For sure it's a socket timeout. I've got about 50 searches in my history lol. I'll fix asap and get the proper PoC in

[simllll]

If it's not the docker setup, what can cause this? It's happening on our side too, but only during a docker build step. We are using Buildkit, but there were no issues before. We also have no issues with other installations steps (e.g. apk or apt-get). Any ideas what could be the reason? I tried to narrow it down, but I'm out of ideas (keepalive, maxsockets and timeout settings dont change anything here..). It randomly fails, no way to reproduce this in a reliable way. Do you all use the alpine node image?

[the-vampiire]

@martinburger @CodeTroopers here is a repo i made to reproduce. it has a script build-test.sh

it will run each build 20 times (starting with no downgrade to 8.3) and exit on the first error.

you can switch the order to try the downgrade tests first.

on my mac it just failed first attempt

running build tests on host [Darwin]
running 20 build attempts of Dockerfile [Dockerfile] tag [npm-cli-socket-bug:no-8.3]
build attempt: [0]
[+] Building 14.3s (9/11)
 => [internal] load build definition from Dockerfile                                0.0s
 => => transferring dockerfile: 413B                                                0.0s
 => [internal] load .dockerignore                                                   0.0s
 => => transferring context: 2B                                                     0.0s
 => [internal] load metadata for docker.io/library/node:17-alpine                   0.4s
 => [internal] load build context                                                   0.1s
 => => transferring context: 1.58MB                                                 0.1s
[+] Building 14.5s (9/11)
 => => transferring context: 1.58MB                                                 0.1s
 => CACHED [prod-dependencies 1/3] FROM docker.io/library/node:17-alpine@sha256:0e  0.0s
 => CACHED [compile 2/3] WORKDIR /compile                                           0.0s
 => [prod-dependencies 2/3] COPY package*.json ./                                   0.2s
[+] Building 73.3s (10/11)
 => [internal] load build definition from Dockerfile                                0.0s
 => => transferring dockerfile: 413B                                                0.0s
 => [internal] load .dockerignore                                                   0.0s
 => => transferring context: 2B                                                     0.0s
 => [internal] load metadata for docker.io/library/node:17-alpine                   0.4s
 => [internal] load build context                                                   0.1s
 => => transferring context: 1.58MB                                                 0.1s
 => CACHED [prod-dependencies 1/3] FROM docker.io/library/node:17-alpine@sha256:0e  0.0s
 => CACHED [compile 2/3] WORKDIR /compile                                           0.0s
 => [prod-dependencies 2/3] COPY package*.json ./                                   0.2s
 => [prod-dependencies 3/3] RUN npm install --only=production                       8.1s
 => [all-dependencies 2/3] COPY --from=prod-dependencies ./ ./                      1.4s
 => ERROR [all-dependencies 3/3] RUN npm install                                   62.3s
------
 > [all-dependencies 3/3] RUN npm install:
#10 25.33 npm WARN deprecated messageformat@2.3.0: Package renamed as '@messageformat/core', see messageformat.github.io for more details. 'messageformat' will eventually provide a polyfill for Intl.MessageFormat, once it's been defined by Unicode & ECMA.
#10 27.71 npm WARN deprecated @types/axios@0.14.0: This is a stub types definition for axios (https://github.com/mzabriskie/axios). axios provides its own type definitions, so you don't need @types/axios installed!
#10 28.22 npm WARN deprecated codecov@3.8.3: https://about.codecov.io/blog/codecov-uploader-deprecation-plan/
#10 61.16 npm ERR! code ERR_SOCKET_TIMEOUT
#10 61.17 npm ERR! network Socket timeout
#10 61.17 npm ERR! network This is a problem related to network connectivity.
#10 61.17 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#10 61.17 npm ERR! network
#10 61.17 npm ERR! network If you are behind a proxy, please make sure that the
#10 61.17 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#10 61.17
#10 61.17 npm ERR! A complete log of this run can be found in:
#10 61.17 npm ERR!     /root/.npm/_logs/2022-02-21T17_23_20_992Z-debug-0.log
------
executor failed running [/bin/sh -c npm install]: exit code: 1

on linux machine failed first attempt

[the-vampiire]

@CodeTroopers @martinburger update on linux machine running the build-test script:

with downgrade to 8.3, success over 20 no-cache builds

running build tests on host [Linux]

running 20 build attempts of Dockerfile [Dockerfile.downgrade] tag [npm-cli-socket-bug:downgrade-8.3]

build attempt: [0]
...
completed 20 build attempts without failure

without downgrade, failed first attempt

running 20 build attempts of Dockerfile [Dockerfile] tag [npm-cli-socket-bug:no-8.3]
build attempt: [0]
Sending build context to Docker daemon  1.591MB
Step 1/9 : FROM node:17-alpine as prod-dependencies
 ---> 025c3cbb849f
Step 2/9 : COPY package*.json ./
 ---> b0a7025cec35
Step 3/9 : RUN npm install --only=production
 ---> Running in 8edee0a0e16d

added 201 packages, and audited 202 packages in 17s

20 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New minor version of npm available! 8.3.1 -> 8.5.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.1>
npm notice Run `npm install -g npm@8.5.1` to update!
npm notice
Removing intermediate container 8edee0a0e16d
 ---> a6e173e6bb46
Step 4/9 : FROM node:17-alpine as all-dependencies
 ---> 025c3cbb849f
Step 5/9 : COPY --from=prod-dependencies ./ ./
 ---> 627f1a851022
Step 6/9 : RUN npm install
 ---> Running in 1f26a6204737
npm WARN deprecated messageformat@2.3.0: Package renamed as '@messageformat/core', see messageformat.github.io for more details. 'messageformat' will eventually provide a polyfill for Intl.MessageFormat, once it's been defined by Unicode & ECMA.
npm WARN deprecated @types/axios@0.14.0: This is a stub types definition for axios (https://github.com/mzabriskie/axios). axios provides its own type definitions, so you don't need @types/axios installed!
npm WARN deprecated codecov@3.8.3: https://about.codecov.io/blog/codecov-uploader-deprecation-plan/
npm ERR! code ERR_SOCKET_TIMEOUT
npm ERR! network Socket timeout
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-02-21T18_38_10_942Z-debug-0.log
The command '/bin/sh -c npm install' returned a non-zero code: 1

[simllll]

I tried to find the root of this, and stumpled upon make-fetch-happen and minipass-fetch .. Well, and there is an issue that sounds very much like ours: https://github.com/npm/make-fetch-happen/pull/100 I guess the freeSocketTimeout fixes some cases, but not all of them.

I wanted to play around by using npm > 8.3 and using an older make-fetch-happen version, but I'm still unable to reproduce this on my side in a way that would make sense. @the-vampiire do you have this running on your local machine? what docker version are you running on? I'm on Docker version 20.10.12, build e91ed57 locally and cannot reproduce the issue on my device, only on our CI/CD pipeline.

[the-vampiire]

@simllll yes reproducing on macOS and linux. Exact same Docker version / build.

Confirmed with the repo I linked above that downgrading to 8.3 consistently (20 no cache builds without error) fixed it.

[simllll]

And can you also reproduce it with latest npm 8.5.1? Just asking because in this version the freesocketimeout has been explicitly set to 15seconds again, which was the case in earlier versions (default value of agentkeepalive). see https://github.com/npm/make-fetch-happen/pull/100#issuecomment-1040529776

[the-vampiire]

@simllll confirmed to resolve!

• downgrade to 8.3.0
• upgrade to 8.5.1

upgrading to npm@8.5.1

bash build-test.sh 10
running [10] build attempts of Dockerfile [Dockerfile.upgrade] [npm-cli-socket-bug:upgrade-8.5.1] on host [Linux]
build attempt: [1]
build attempt: [1] completed without failure
build attempt: [2]
build attempt: [2] completed without failure
build attempt: [3]
build attempt: [3] completed without failure
build attempt: [4]
build attempt: [4] completed without failure
build attempt: [5]
build attempt: [5] completed without failure
build attempt: [6]
build attempt: [6] completed without failure
build attempt: [7]
build attempt: [7] completed without failure
build attempt: [8]
build attempt: [8] completed without failure
build attempt: [9]
build attempt: [9] completed without failure
build attempt: [10]
build attempt: [10] completed without failure
completed [10] build attempts without failure

[martinburger]

I just run:

❯ npm -v
8.4.1

❯ npm update npm -g

❯ npm -v
8.5.1

❯ npm uninstall -g @aws-amplify/cli

❯ npm install -g @aws-amplify/cli

Well, installation worked liked a charm after upgrading NPM to version 8.5.1.