npm hangs under heavy network load on some connections

[busches]

I'm using npm 2.7.4 with nodejs v0.12.2 on Windows 7x64 and I'm running into an odd issue. With the below package.json, when running npm install the process will hang after it runs node-gyp for ws for karma. But if I run npm install -ddd then it doesn't hang and installs correctly.

{
  "name": "npmBug",
  "version": "0.0.0",
  "dependencies": {},
  "devDependencies": {
    "browser-sync": "~1.7.1",
    "karma": "^0.12.31"
  }
}

If I run with just npm install -dd then it hangs at this line:

npm info install karma@0.12.31
npm info postinstall karma@0.12.31
npm verb unlock done using C:\Users\buschs1\AppData\Roaming\npm-cache\_locks\karma-f090bd27423c129a.lock for C:\Users\buschs1\Desktop\npmBug\node_modules\karma

I'm able to reproduce this on another machine (same setup, node, Win7x64, but doesn't have python, etc. setup for node-gyp) and it happens when using io.js v1.6.3 as well. Since it does install when changing logging settings in npm, I've opened the issue against npm, let me know if it belongs elsewhere. Please let me know if you need anymore information.

[smikes]

Thanks for the report. It sounds like it could be a race condition in the gyp or related code.

[busches]

I was able to test this on a linux box yesterday and had zero issues FYI.

[smikes]

Is this still a problem for you?

Could you please upload your npm-debug.log as a gist http://gist.github.com? If possible, run npm with at least one -d flag to increase the logging level.

There have been a lot of improvements to npm -- especially involving git dependencies -- since 2.7. Can you try updating your npm installation?

There is a bad interaction between two known bugs — one in node@>0.11 and iojs and the other in npm@<2.8.2. This can cause ECONNRESET and ETIMEDOUT errors. The full writeup is here: https://github.com/npm/npm/issues/7699 You can fix this problem by updating your npm to the latest (see below).

To update npm on Windows, follow the instructions here: https://github.com/npm/npm/wiki/Troubleshooting#upgrading-on-windows

We are trying to clean up older npm issues, so if we don't hear back from you within a week, we will close this issue. (Don't worry -- you can always come back again and open a new issue!)

Thanks!

[busches]

I do not currently have access to the original PC that encountered this error. I am unable to reproduce the issue on Windows 8.1 x64, with node v0.12.2 and npm 2.7.4 or on a clean VM of Windows 7 x64 with the same node/npm versions. The original PC had Python and MSVS setup to compile ws, etc., so maybe the problem is coming from the interaction with node-gyp. I should be able to retest it on the original PC next week.

[busches]

I was able to recreate it and it actually hangs on both -dd and -ddd now.

Node: v0.12.2 npm: v2.9.0 Windows 7x64 Python: Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)] on win32

Simple package.json: https://gist.github.com/busches/9e03ce80e9fcaac03f3f Log with -dd: https://gist.github.com/busches/e754d0e1b5046766b2e0 Log with -ddd: https://gist.github.com/busches/bdac3d9b101d308b78b2

Complex package.json: https://gist.github.com/busches/82fba2c02d5c50ddb8be Log with -dd: https://gist.github.com/busches/7eaa668ca621c4391a59 Log with -ddd: https://gist.github.com/busches/a29c3ec19d698bae0d87

Each log file hangs on the final step; times waited varied from five minutes to over an hour depending on what I was doing.

Note, I'm using the frontend-maven-plugin to test this, as I'm unable to update the local npm install on the machine currently and it provides an isolated test envrioment. I was getting the error before with the normal npm install as well as with the plugin previously.

Let me know if you need anymore system information or logs.

[busches]

Admin privileges were restored to the faulty machine and I was able to update npm to 2.9.1. I am once again seeing the install pass with -ddd and hang with -dd. Logs and information below.

Node: v0.12.2 npm: v2.9.1 Windows 7x64 Python: Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)] on win32

Simple package.json: https://gist.github.com/busches/9e03ce80e9fcaac03f3f Log with -dd: https://gist.github.com/busches/9c3782a9edf8f9b760c1 Log with -ddd: https://gist.github.com/busches/e8e456ebe113ede5db8a

Thanks.

[smikes]

Thanks for the detailed information. I hope to have a chance to look at it this weekend or early next week.

[jsoverson]

This is happening regularly for me and is unrelated to log level. Module installs regularly, though inconsistently, fail with the "unlock" line being the last line of output, eg

npm info install browserify@10.1.2
npm info postinstall browserify@10.1.2
npm verb unlock done using [homedir]/.npm/_locks/browserify-3eb7ac69b59fdbcb.lock for [install path]/node_modules/browserify

Retries often work, but the failures are very common. Also seen it on venus, socket.io, and karma.

$ uname -a
Darwin egon 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
$ npm -v
2.9.0
$ node -v
v0.12.0

Edit: to add, this is OSX so the windows label should probably be removed. This is occurring on a number of OSX machines in the office (with & without proxy)

[jsoverson]

With npm 2.10.0 I am also seeing installations hang after the following logging output:

npm sill cache afterAdd karma@0.12.32
npm verb afterAdd [homedir]/.npm/karma/0.12.32/package/package.json not in flight; writing
npm verb afterAdd [homedir]/.npm/karma/0.12.32/package/package.json written

(happens on modules other than karma)

I'm still navigating npm code, but i'm seeing cases where cb in asyncMap never gets "exhausted" so does not calling the original callback.

[othiym23]

@jsoverson It sounds like you may be running into an issue that's unrelated to the original issue.

What does uname -n return? If it's 256, you might want to run uname -n 2560 and see if things improve. Apple, for unknowable fruit-related reasons, nerfed the setting for max file descriptors in 10.10.3, and it causes these intermittent lockups on more complicated npm installs. There may be an issue in graceful-fs that's leading to livelocking in npm, but to date, everyone who's seen this issue has been able to fix it by boosting the number of available file descriptors.

[vshabelnyk]

The same problem for me, its hang with some job and spinner after npm verb unlock done using in a lot of different versions of node and npm on linux. But no luck with -ddd unfortunately

[jsoverson]

@othiym23 ulimit -n returned 2048, I bumped it to 5096 and immediately (first run) had npm hang after running the following install with arbitrary modules

npm install grunt karma venus browserify karma mocha jasmine plato

Npm log output right before the hang:

npm verb etag http://registry.npmjs.org/plato from cache
npm verb get saving plato to /Users/u692/.npm/registry.npmjs.org/plato/.cache.json
npm sill addNameRange number 2 { name: 'plato', range: '*', hasData: true }
npm sill addNameRange versions [ 'plato',
npm sill addNameRange   [ '0.1.0',
npm sill addNameRange     '0.2.0',
npm sill addNameRange     '0.3.0',
npm sill addNameRange     '0.3.1',
npm sill addNameRange     '0.4.0',
npm sill addNameRange     '0.4.1',
npm sill addNameRange     '0.4.2',
npm sill addNameRange     '0.4.3',
npm sill addNameRange     '0.4.4',
npm sill addNameRange     '0.4.5',
npm sill addNameRange     '0.4.6',
npm sill addNameRange     '0.4.7',
npm sill addNameRange     '0.5.0',
npm sill addNameRange     '0.6.1',
npm sill addNameRange     '0.6.2',
npm sill addNameRange     '1.0.0',
npm sill addNameRange     '1.0.1',
npm sill addNameRange     '1.1.0',
npm sill addNameRange     '1.2.0',
npm sill addNameRange     '1.2.2',
npm sill addNameRange     '1.3.0',
npm sill addNameRange     '1.4.0' ] ]
npm sill addNamed plato@1.4.0
npm verb addNamed "1.4.0" is a plain semver version for plato
npm sill cache afterAdd plato@1.4.0
npm verb afterAdd /Users/u692/.npm/plato/1.4.0/package/package.json not in flight; writing
npm verb afterAdd /Users/u692/.npm/plato/1.4.0/package/package.json written

[thomassuckow]

Node: 0.12.1 NPM: 2.10.0 Centos6 (Docker container)

[ERROR] npm info install browser-sync@2.5.3
[ERROR] npm info postinstall browser-sync@2.5.3
[ERROR] npm verb unlock done using /home/jenkins/.npm/_locks/browser-sync-4d11b0e54cac4766.lock for /home/jenkins/workspace/Project/node_modules/browser-sync

[mischkl]

I'm also getting issues with NPM hanging at random points in the installation process. Usually it happens right after postinstall; using -ddd I also saw "unlock done" before the hang. This is on Mac OS X 10.9.5 with Node 0.12.0 and NPM 2.10.1 (but also experienced lots of random hanging with the previous couple of NPM versions).

[Endellur]

I am seeing this issue with Mac 10.10.3 (already checked and updated max files), node 0.12.4 and npm 2.10.1. npm install will work sometimes if I use -verbose but anything else hangs indefinitely. It seems to always stop after unlocking after the socket.io postinstall if that helps.

[rogerz]

I'm getting this issue while installing gulp-duojs on MacOS 10.10.3.

It hangs on

npm info install duo-test@0.4.0
npm info postinstall duo-test@0.4.0
npm verb unlock done using /Users/zhangyujun/.npm/_locks/duo-test-fdcca65dddf451b9.lock for /Users/zhangyujun/Workspace/we-sign-app/node_modules/gulp-duojs/node_modules/duo/node_modules/duo-test

Resolved with npm cache clean and re-install

[vshabelnyk]

Unfortunately clean dont help on my machine. And also the same issue all the time happens on machines of all my dev team. Only ci server not affected. I think there is quite old installation of node and npm.

[othiym23]

It sounds like there's probably some kind of subtle race condition at play here (for everyone's information, changing the loglevel doesn't actually change which code gets executed by npm, but does subtly affect the timing of things because of the I/O it schedules, which is part of what leads me to believe a race condition is happening here), but since all that we've managed to document thus far is its effects, I don't really have a very good idea where to look to fix it. My guess is that there's something happening inside the cache, perhaps with lockfiles, but I don't know for sure.

It doesn't sound like any of you have this problem 100% reproducibly, but if the package.json files you're using are shareable, putting them in a gist and linking to it here would be helpful. If you can get this to happen consistently every time, please tell me what you're doing, what versions of Node and npm you're running, and as much detail about which operating system, distribution, and runtime environment as you can scrape up.

[thomassuckow]

https://gist.github.com/thomassuckow/b1e0b38dc31fea726d11

[busches]

@othiym23 I was able to recreate it with the complex.json from above, but no longer the simple one.

I waited ten minutes and saw no CPU usage or memory usage changes by node.exe before declaring it hung on this line:

npm verb unlock done using C:\Users\buschs1\AppData\Roaming\npm-cache\_locks\gulp-protractor-b8d053875c79eb1b.lock for C:\Users\buschs1\Desktop\npmBug\node_modules\gulp-protractor

Complex package.json: https://gist.github.com/busches/82fba2c02d5c50ddb8be Log with -dd: https://gist.github.com/busches/99b951fbf942b25928c8

Node: v0.12.4 npm: v2.11.0 Python: Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)] on win32 VS Express 2012 for Desktop: 12.0.31101.00 Update 4

System info:

OS Name:                   Microsoft Windows 7 Professional
OS Version:                6.1.7601 Service Pack 1 Build 7601
System Manufacturer:       Hewlett-Packard
System Model:              HP EliteBook Folio 9470m
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~2301 Mhz
BIOS Version:              Hewlett-Packard 68IBD Ver. F.32, 12/5/2012
Total Physical Memory:     8,056 MB

Let me know if you need any more information.

Thanks!

[kilianc]

I am experiencing a similar issue when running npm in a docker container with debian or ubuntu.

No error messages but sometimes, inconsistenlty a postinstall script or node-gyp hangs. Happened with handlebar, request, but the one I see it hanging the most is socket.io.

Below the Dockerfile to reproduce:

FROM google/debian:wheezy
MAINTAINER me@nailik.org

# pkgs
RUN apt-get update && apt-get install --no-install-recommends -y -q \
  build-essential \
  ca-certificates \
  curl \
  gcc \
  git \
  graphicsmagick \
  locales \
  make \
  python \
  tesseract-ocr \
  tesseract-ocr-eng

RUN apt-get clean

# locales
RUN dpkg-reconfigure locales && \
  locale-gen C.UTF-8 && \
  /usr/sbin/update-locale LANG=C.UTF-8

ENV LC_ALL C.UTF-8

# node
ENV NODE_VERSION 0.12.4
ENV PATH $PATH:/node/bin
RUN mkdir /node && \
  curl --silent http://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.gz | \
  tar xvzf - -C /node --strip-components=1
RUN node --version && npm --version

# npm
RUN npm install -g -d bower
RUN npm install -g -d gulp
RUN npm install -g -d rtail

To add more info, removing make, gcc, locales and make seemed to fix it consistenly. Here the working one:

# boilerpale
FROM debian:jessie
MAINTAINER me@nailik.org

ENV DEBIAN_FRONTEND noninteractive

# pkgs
RUN apt-get update && apt-get install --no-install-recommends -y -q \
  build-essential \
  ca-certificates \
  ghostscript \
  git \
  curl \
  graphicsmagick \
  tesseract-ocr \
  tesseract-ocr-eng \
&& apt-get clean

# node
ENV NODE_VERSION 0.12.4
ENV PATH $PATH:/node/bin
RUN mkdir /node && \
  curl --silent http://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.gz | \
  tar xvzf - -C /node --strip-components=1

# npm
RUN npm install -g -d \
  bower \
  gulp \
  rtail && \
  npm cache clean

[denouche]

Hi

I encounter the exactly same problem.

I'm trying to run a npm install in a docker container, I tried both debian:8.1 and ubuntu:14.04.2 and I have the same result.

My host is running a debian 8.

npm info using npm@2.10.1
npm info using node@v0.12.4

When the hangs come I can kill the process by Ctrl+C, and retrying the same npm install without doing anything else juste run fine.

For me, the module that ofter hangs is:

npm info linkStuff mongoose@4.0.5
npm sill linkStuff mongoose@4.0.5 has /home/jenkins/tmp/node_modules as its parent node_modules
npm verb linkBins mongoose@4.0.5
npm verb linkMans mongoose@4.0.5
npm verb rebuildBundles mongoose@4.0.5
npm verb rebuildBundles [ 'async',
npm verb rebuildBundles   'bson',
npm verb rebuildBundles   'hooks-fixed',
npm verb rebuildBundles   'kareem',
npm verb rebuildBundles   'mongodb',
npm verb rebuildBundles   'mpath',
npm verb rebuildBundles   'mpromise',
npm verb rebuildBundles   'mquery',
npm verb rebuildBundles   'ms',
npm verb rebuildBundles   'muri',
npm verb rebuildBundles   'regexp-clone',
npm verb rebuildBundles   'sliced' ]
npm info install mongoose@4.0.5
npm info postinstall mongoose@4.0.5
npm verb unlock done using /home/jenkins/.npm/_locks/mongoose-4113ef694315d76d.lock for /home/jenkins/tmp/node_modules/mongoose

Do you have any information to help me to run my npm install successfully ?

Thank you

Edit: To be able to compile some node modules, I installed the following packages on the debian:8.1 and ubuntu:14.04.2 containers:

apt-get update
apt-get install build-essential libkrb5-dev curl python
curl -sL https://deb.nodesource.com/setup_0.12 | bash -
apt-get install nodejs
npm install -g npm

[shaunthomas999]

+1. I am facing the same problem as mentioned in previous 2 comments while trying to install npm packages in docker container.

[SirAnthony]

node 0.12.4, npm 2.10.1 node-inspector install -g hangs with same issue:

npm verb about to build /usr/lib/node_modules/node-inspector/node_modules/yargs
npm info build /usr/lib/node_modules/node-inspector/node_modules/yargs
npm info linkStuff yargs@3.11.0
npm verb linkBins yargs@3.11.0
npm verb linkMans yargs@3.11.0
npm verb rebuildBundles yargs@3.11.0
npm verb rebuildBundles [ 'camelcase', 'cliui', 'decamelize', 'window-size' ]
npm info install yargs@3.11.0
npm info postinstall yargs@3.11.0
npm verb unlock done using /home/sergey/.npm/_locks/yargs-2246090c8de8b128.lock for /usr/lib/node_modules/node-inspector/node_modules/yargs

It is different modules every time, not only yargs: which, ws, or any other module which is installed at the end.

[brandonros]

Having a very similar problem trying to install browserify/literalify. Can't figure out exactly where it hangs each time even with logging/debugging.

Edit: I had success by breaking the install comand into two separate commands.

[barraq]

Same issue as @kilianc and @denouche

# Based on ruby 2.1.5
FROM ruby:2.2.1-wheezy

RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8

ENV PG_MAJOR 9.4
ENV PG_VERSION 9.4.1-1.pgdg70+1

RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list

# Update and install dependencies
RUN apt-get update -qq \
    && apt-get install -y curl \
    && curl -sL https://deb.nodesource.com/setup | bash - \
    && apt-get install -y build-essential apt-utils libpq-dev \
                          nodejs npm \
                          postgresql-client-$PG_MAJOR

# Install utilities
RUN npm install -dd -g bower

# Copy Gemfile and install bundle
WORKDIR /tmp
COPY Gemfile Gemfile
COPY Gemfile.lock Gemfile.lock
RUN bundle install

# Setup app folders
RUN mkdir /kairos
ADD . /kairos
WORKDIR /kairos

[denouche]

After a loooot of tests, my secret recipe to make it works is :

• Nodejs 0.10.39
• npm 2.11.3 (you can upgrade using (npm install -g npm)
• node-gyp installed in global (npm install -g node-gyp)
• just a little update for the fun (npm update)

With this I'm able to run my npm install

You can find my Dockerfile here: https://github.com/denouche/docker-jenkins-agents/blob/master/nodejs-0.10.x/Dockerfile

I found help in this post: http://gangmax.me/blog/2013/05/13/resolve-npm-update-node-gyp-hung-problem/

It's still hangs sometimes but I think that it's ok 80% of time.

Tell me if it's okay with that for you?

[tomwang1013]

Same issue when installing gitbook: sudo npm install -g --verbose gitbook it happened everytime with the last message: npm verb unlock done using /home/tomwang/.npm/_locks/npm-0835e8134ded4970.lock for /usr/local/lib/node_modules/gitbook/node_modules/npm

os: ubuntu 14.04 npm: '2.11.2', node: '0.12.5'

[mmarod]

I'm in the same boat as @denouche ... It works about 80% of the time and when it doesn't I just re-run the build and it seems to work. In addition to his steps I ensure that I have set ulimit -S -n 2560 before running.

[vshabelnyk]

I have a feeling that something unites all of us, how about using network connection trough proxy? Or using private npm registry with caching proxy like nexus? Because i and all my coworkers faced this issues on different os and node/npm. But i dont have this issues on home machine.

[mischkl]

@RootMale maybe you're onto something - we are also using an in-house registry with caching proxy (Artifactory).

[denouche]

For me:

• No proxy.
• No private npm registry

The only notable things in my case are:

• I'm using Docker
• My Docker host is a Debian 8.1 running in a Virtualbox VM.

[jsoverson]

FWIW I installed npm 3.0.0 beta and am still seeing npm hang, although this time it is stalling after shrinkwrap logging, eg

npm sill addShrinkwrap Adding shrinkwrap to esutils
npm sill addShrinkwrap Adding shrinkwrap to detect-indent
npm sill addShrinkwrap Adding shrinkwrap to repeating
npm sill addShrinkwrap Adding shrinkwrap to strip-json-comments
npm sill addShrinkwrap Adding shrinkwrap to regenerator

The command i'm using to test is:

for i in `seq 1 100`; do rm -rf node_modules/ && npm install browserify babel eslint jshint plato karma watchify; done

$ node -v
v0.12.4
$ npm -v
3.0.0
$ uname -a
Darwin egon 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
$ ulimit -n
2560

[BlueHotDog]

same here :( :+1:

[denouche]

Does everybody has this problem in a docker container, or it is on a regular host for you? Le 3 juil. 2015 4:02 PM, "Danni Friedland" notifications@github.com a écrit :

same here :( [image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/npm/npm/issues/7862#issuecomment-118359415.

[varunpalekar]

+1. Same here : (

[Endellur]

I am not using a docker container but I am behind a proxy. I don't use a private npm registry. The problem has gotten worse, however, and now it no longer works reliably using debug. I only have an issue on Mac, using Linux (SL6) behind the same proxy I have no problems.

[BlueHotDog]

i'm on a docker container, a day passed without being able to do any work..

please please fix, or provide some sort of fix.

[denouche]

Did you try with these steps?

• Nodejs 0.10.39
• npm 2.11.3 (or 2.12) (you can upgrade using (npm install -g npm)
• node-gyp installed in global (npm install -g node-gyp)
• just a little update for the fun (npm update)

[denouche]

You can see my dockerfile here https://github.com/denouche/docker-jenkins-agents/blob/master/nodejs-0.10.x/Dockerfile

[BlueHotDog]

@denouche i'm on node 0.12(cannot downgrade) but will try your other steps..

[denouche]

This is not solving the problem, with this it works for me about 80% of times. But it's a workaround while waiting for a fix.

[kilianc]

We should gather all cases and dockerfiles and create a test suite! How do you guys feel about creating a repo for this? It would be nice to at least have a sign from the maintainers!

[BlueHotDog]

this makes docker virtually unusable for node.. :(

[shaunthomas999]

Sharing my experience: I had severe hanging issues with npm install with Docker version 1.6.2. Docker has released version 1.7.0 on 2015-06-16. Npm install is a lot better with this version.

Npm install hangs in VPN connection with Docker version 1.7.0 as well.

[vshabelnyk]

Problem persist in both cases with Docker or without it, bot in both cases behind proxy.

[vshabelnyk]

Can confirm that problem gone without proxy. But we should use it due to corporate rules. However no issues thrown during install with -ddd.

[vshabelnyk]

UPD. Log output after hang: npm sill cache afterAdd tough-cookie@2.0.0 npm verb afterAdd /home/shav/.npm/tough-cookie/2.0.0/package/package.json not in flight; writing npm verb afterAdd /home/shav/.npm/tough-cookie/2.0.0/package/package.json written

Almost always we get infifnite spinner after "afterAdd" or "unlock done".

[thomassuckow]

Hangs without an explicit proxy. Though we do have some kind of security appliance between us and the internet. Tried removing packages that use socket.io (avoid old ws package) but no joy. Tried with and without an onsite repository mirror.

Always hangs for me on "[ERROR] npm verb unlock done using ..." even with "silly" debug level There is a lock file present in the ~/.npm for a package to be installed. Maybe it is waiting for it to go away?

npm@2.7.1 node@v0.12.1

[varunpalekar]

I am also facing this problem, but now I am using http registry insted of https registry. You can also change by this:

npm config set registry http://registry.npmjs.org/

Hope this will help you. Thanks.