Reinstalling npm v3 fails on Docker.

[statico]

Update 2016-09-06 with temporary fix

Fix from @davidbarton

# Fix bug https://github.com/npm/npm/issues/9863
RUN cd $(npm root -g)/npm \
  && npm install fs-extra \
  && sed -i -e s/graceful-fs/fs-extra/ -e s/fs\.rename/fs.move/ ./lib/utils/rename.js

Alternate fix from @nordluf

RUN npm install --production
RUN mv ./node_modules ./node_modules.tmp && mv ./node_modules.tmp ./node_modules && npm install

Background

Our project initialization scripts make sure a specific version of npm is installed. If npm 3.x is accidentally installed twice the install fails with EXDEV: cross-device link not permitted. Installing additional modules results in the same error.

npm 2.x does not have this problem.

Using Docker 1.8.2, this Dockerfile:

FROM ubuntu:14.04

ENV NPM_CONFIG_LOGLEVEL info
ENV NODE_VERSION 4.1.1

RUN apt-get -y install curl

RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz" \
  && tar -xzf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local --strip-components=1 \
  && rm "node-v$NODE_VERSION-linux-x64.tar.gz"

RUN npm install -g npm@3
RUN npm install -g npm@3 # again!
RUN npm install -g coffee-script

CMD [ "/bin/bash" ]

Results in the following:

$ docker build -t npmtest ~/misc/npmtest
Sending build context to Docker daemon 2.048 kB
Step 0 : FROM ubuntu:14.04
 ---> 91e54dfb1179
Step 1 : ENV NPM_CONFIG_LOGLEVEL info
 ---> Using cache
 ---> 8794a99ac0a2
Step 2 : ENV NODE_VERSION 4.1.1
 ---> Using cache
 ---> 3fb65b4c7d55
Step 3 : RUN apt-get -y install curl
 ---> Using cache
 ---> e1a9f7308089
Step 4 : RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz"   && tar -xzf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local --strip-components=1   && rm "node-v$NODE_VERSION-linux-x64.tar.gz"
 ---> Using cache
 ---> 942c49d39079
Step 5 : RUN npm install -g npm@3
 ---> Using cache
 ---> 871d339a3944
Step 6 : RUN npm install -g npm@3 # again!
 ---> Running in 55fbedab36cf
npm info it worked if it ends with ok
npm info using npm@3.3.5
npm info using node@v4.1.1
npm info attempt registry request try #1 at 6:46:54 PM
npm http request GET https://registry.npmjs.org/npm
npm http 200 https://registry.npmjs.org/npm
npm info lifecycle npm@3.3.5~preinstall: npm@3.3.5
npm ERR! Linux 4.0.9-boot2docker
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "npm@3"
npm ERR! node v4.1.1
npm ERR! npm  v3.3.5
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! code EXDEV
npm ERR! errno -18
npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /npm-debug.log
The command '/bin/sh -c npm install -g npm@3 # again!' returned a non-zero code: 238

[whummer]

+1, when trying to install gulp twice:

npm ERR! Linux 3.19.0-30-generic
npm ERR! argv "node" "/usr/local/bin/npm" "install" "--ignore-scripts" "-g" "gulp@^3.9.0"
npm ERR! node v0.12.7
npm ERR! npm  v3.3.6
npm ERR! path /usr/local/lib/node_modules/gulp
npm ERR! code EXDEV
npm ERR! errno -18

npm ERR! EXDEV, rename '/usr/local/lib/node_modules/gulp'
npm ERR! 

[SteveMarshall]

This appears to be because aufs (Docker's default filesystem) will often cause rename(2) to return EXDEV, regardless of apparent aufs device boundaries. From the aufs manual:

To rename(2) directory may return EXDEV even if both of src and tgt are on the same aufs. When the rename-src dir exists on multiple branches and the lower dir has child(ren), aufs has to copyup all his children. It can be recursive copyup. Current aufs does not support such huge copyup operation at one time in kernel space, instead produces a warning and returns EXDEV. Generally, mv(1) detects this error and tries mkdir(2) and rename(2) or copy/unlink recursively. So the result is harmless. If your application which issues rename(2) for a directory does not support EXDEV, it will not work on aufs. Also this specification is applied to the case when the src directroy exists on the lower readonly branch and it has child(ren).

The actual error in npm appears to be during finalize for a new version of an already-installed module, when moving the old version to a .DELETE folder.

Based on the comment above from the aufs documentation, that would suggest that calls to fs.rename should probably be slightly more carefully handled, possibly warranting wrapping in graceful-fs?

[othiym23]

graceful-fs is already used consistently throughout npm, but since it's mostly intended to deal only with EMFILE issues, it isn't going to help here. I'm pretty sure that npm's behavior here is, in general, what we want it to be doing to satisfy the goal of more robust, transactional installs, but I wouldn't object to a patch that worked more consistently with the grain of aufs and Docker in general.

[qrpike]

This happens to me when running npm update. Since docker caches my NPM install, I only run npm update to check for newer packages, not re-installing everything.

It's very frustrating. NPM3 is so much better, but it's killing my docker images because of this 1 small issue.

Any way we can get a fix for this?

[iarna]

@qrpike The fastest way to get a fix for this would be to write a module that implements an fs.rename that, if it gets EXDEV falls back to making the new directory and recursively calling itself on all of the contents of the original dir and rmdiring the original before returning. If an extensively tested module like I've described existed, the patch to npm would be straightforward and I'd be happy to prioritize its use.

My biggest concern (and one I haven't investigated yet) is that ordinarily rename is an atomic operation, in fact, it's one of the only atomic operations one has available when dealing with the filesystem. If the code calling it is expecting it to be atomic, we wouldn't be able to use a module as I've described.

[mhart]

Oh ouch, just ran into this too – it's completely preventing npm3 from being updated on docker images.

Reproduce:

docker run mhart/alpine-node:5 npm install -g npm
# OR
docker run mhart/alpine-node:5 npm update -g npm

[jsmith-dev]

+1

npm ERR! Linux 4.0.5
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "i" "-g" "npm"
npm ERR! node v5.0.0
npm ERR! npm  v3.3.6
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! code EXDEV
npm ERR! errno -18
npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE'
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /pipeline/source/npm-debug.log

[amir-rahnama]

+1

Step 7 : RUN npm install http-server
 ---> Running in 47faed65875d
npm info it worked if it ends with ok
npm info using npm@3.3.6
npm info using node@v5.0.0
npm info attempt registry request try #1 at 8:48:06 AM
npm http request GET https://registry.npmjs.org/http-server
npm http 304 https://registry.npmjs.org/http-server
npm info attempt registry request try #1 at 8:48:06 AM
npm http request GET https://registry.npmjs.org/colors
npm http 304 https://registry.npmjs.org/colors
npm info lifecycle http-server@0.8.5~preinstall: http-server@0.8.5
npm ERR! Linux 4.1.10-boot2docker
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "http-server"
npm ERR! node v5.0.0
npm ERR! npm  v3.3.6
npm ERR! path /app/node_modules/http-server
npm ERR! code EXDEV
npm ERR! errno -18
npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/app/node_modules/http-server' -> '/app/node_modules/.http-server.DELETE'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /app/npm-debug.log```

[dieend]

Having this same problem. Any workaround?

[0x6a68]

use docker image node:4 which uses npm 2.14.7

[haggholm]

@iarna node-fs-extra contains a move() operation which attempts rename() and uses other mechanisms on errors like EXDEV.

My biggest concern (and one I haven't investigated yet) is that ordinarily rename is an atomic operation, in fact, it's one of the only atomic operations one has available when dealing with the filesystem. If the code calling it is expecting it to be atomic, we wouldn't be able to use a module as I've described.

Knowing that this issue is a major stumbling block for Docker containers in particular, could you consider a conditional based on e.g. checking if npm is running in a container, or an environment variable like I_ACCEPT_THE_RISK_OF_NONATOMIC_MOVES? —using fs.rename() by default, but fs-extra’s move() if enabled. (Of course, I say this hoping that an opt-in behaviour, being less disruptive, could make it in sooner….)

[haggholm]

Added trivial PR to go with previous comment. I don’t know if that’s the sort of thing you accept for npm, so feel 100% free to toss it with no risk of thereby offending me… Unfortunately I can’t test it properly as my Docker environment is also suffering from #10372 (not really related except for breaking in the same Docker setup).

[armandabric]

My CI is also blocked by this issue.

[trygve-lie]

Did run into the same issue on a plain npm install. Node 5.1.0 and the bundled npm version.

Switching the Docker image from aufs to overlayfs seems to work.

[jakutis]

Happens to me also on docker image (Dockerfile FROM node:5.1.0) build:

npm ERR! EXDEV: cross-device link not permitted, rename '/app/node_modules/sails-permissions/node_modules/lodash' -> '/app/node_modules/sails-permissions/node_modules/sails-generate-entities/node_modules/lodash'

Although FROM node:5.0.0 works perfectly..

[johandry]

I use a workaround to upgrade npm that is not failing. Instead of 'npm install -g npm' I use:

curl -L https://npmjs.org/install.sh | sh

My Dockerfile have:

RUN     yum install -y epel-release
RUN     yum install -y nodejs npm

# Upgrade node and npm to latest version
RUN     npm cache clean
RUN     npm install -g n
RUN     n stable
RUN     curl -L https://npmjs.org/install.sh | sh

The result is:

[root@a7923d2645b4 ~]# node -v
v5.0.0
[root@a7923d2645b4 ~]# npm -v
3.5.0

[cookandy]

Same issue when trying to downgrade npm from a newer version of node:

npm ERR! Linux 4.1.10-boot2docker
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "npm@2.14.13"
npm ERR! node v5.1.0
npm ERR! npm  v3.3.12
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! code EXDEV
npm ERR! errno -18
npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE'

Dockerfile:

FROM node:5.1.0

ENV NPM_VERSION 2.14.13

RUN  npm install -g npm@"$NPM_VERSION"  \
      && npm cache clear

CMD [ "node" ]

https://github.com/nodejs/docker-node/issues/70

[iilei]

same issue with node:5.1.0

[parkr]

Also having this problem with Node 5.0.0 and NPM 3.3.6.

[TomTomaso]

same problem here: pm ERR! Linux 4.1.13-boot2docker npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "npm@3.3.6" npm ERR! node v5.0.0 npm ERR! npm v3.3.6 npm ERR! path /usr/local/lib/node_modules/npm npm ERR! code EXDEV npm ERR! errno -18 npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE' npm ERR! npm ERR! If you need help, you may report this error at: npm ERR! https://github.com/npm/npm/issues

npm ERR! Please include the following file with any support request: npm ERR! /npm-debug.log The command '/bin/sh -c npm install -g npm@"$NPM_VERSION"' returned a non-zero code: 238

[dciccale]

Same issue:

npm     ERR!   Linux 3.13.0-40-generic
npm     ERR!     argv   "/usr/local/bin/node" "/usr/local/bin/npm" "update" "-g" "npm"
npm     ERR!     node   v5.2.0
npm     ERR!     npm    v3.3.12
npm     ERR!     path   /usr/local/lib/node_modules/npm
npm     ERR!     code   EXDEV
npm     ERR!     errno   -18
npm     ERR!     syscall   rename

npm     ERR!   EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE'
npm     ERR!   
npm     ERR!   If you need help, you may report this error at:
npm     ERR!       <https://github.com/npm/npm/issues>

npm     ERR!   Please include the following file with any support request:
npm     ERR!       /home/app/npm-debug.log

[evelynharthbrooke]

Have you guys tried updating your guys' npm versions to 3.5.2?

[evelynharthbrooke]

@dciccale Also, why are you using such an old kernel?

[dciccale]

@KamranMackey That is the log output from dockerhub

[frank-dspeed]

Simply replace the rename method with something cross device compatible the way to go is fork this of and implament https://www.npmjs.com/package/fs.extra as replacement for fs methods then use move not fs.rename

[haggholm]

@frank-dspeed Like my PR from November 12?

[frank-dspeed]

your pr was not so good i have done a working one https://github.com/npm/npm/issues/10805

they used graceful fs already so i choosed its extra npm added it all rewrited the rename.js and tested it with docker because i am from docker dev and work much with nodejs i was interrested in fixing this.

i needed it to test my nodejs moduls in docker containers by mounting dev moduls into production like containers.

[shellscape]

+1 with respect to the busy lives the maintainers lead, I'm surprised this hasn't been given more priority as it's a change in a major version that is pervasively impacting a major (and quite common these days) ecosystem negatively.

[othiym23]

This hasn't been given a higher priority for two reasons:

1. the team is presently focused on other things, namely the long-overdue work of getting npm's test suite in good enough shape that it's a trustworthy metric of product quality
2. the proposed solutions thus far are inadequate – the whole reason for that final rename is to eliminate a race condition, and to work towards making the install process more atomic. Switching it with a move that copies on EXDEV defeats that purpose. An alternative might be to do all the work in a subdirectory of the destination and then hoist things up as part of the install's finalization process, but that's a solution very specific to Docker, and probably poses a different set of complications for other users (and also nobody's verified that that would work any better for Docker).

I would like to see this problem resolved, but it needs to be done in such a way that both goes with the grain of how Docker wants to be used, and without breaking npm for its other users.

[frank-dspeed]

patched npm version aviable at http://github.com/DIREKTSPEED-LTD/npm master branch trys to keep current. to original npm also see #10805 if any one interested

[IntellixWeb]

@frank-dspeed what branch / tag should i look for ?

[frank-dspeed]

master works great

[frank-dspeed]

example dockerfile:

FROM node:5.1
MAINTAINER Frank Lemanschik <frank@dspeed.eu>

ENV PORT 80
# pm2-gui start 8090
# ADD /dscms /app
# RUN git clone http://github.com/anyrepo /app
# ADD /app
WORKDIR /app

EXPOSE 80
ENV NODE_ENV=production
RUN npm install -g https://github.com/DIREKTSPEED-LTD/npm
RUN npm install

##
#RUN npm install $(ls node_modules | grep dscms | grep -v domainSettings)
ENTRYPOINT [ "pm2" ]
CMD [ "start", "/app/bin/www", "-i", "0", "--no-daemon", "--watch"]

[IntellixWeb]

@frank-dspeed Sounds great. Trying now and will get back.

[frank-dspeed]

Opening Issus on that fork as you see in #10805 its not planed to move that changes in so i will maintain a external fork with own issus for the docker users.

[IntellixWeb]

@frank-dspeed Still doesn't do it for me. Do I need to change the storage-driver?

Here's my Dockerfile:

############################################################
# Dockerfile for Sane-stack to run sails.js API application
############################################################

FROM node:5.1
MAINTAINER Intellix <office@intellix.eu>

WORKDIR /var/www/sails/projects/testProject 

RUN npm install -g https://github.com/DIREKTSPEED-LTD/npm
RUN npm install -g grunt bower pm2 sails
RUN npm install sails-mongo  --save

ENTRYPOINT ["pm2", "start", "app.js", "--no-daemon"]
# Expose ports.
EXPOSE 1337

And this is what I'm getting:

npm ERR! Linux 3.19.0-42-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "https://github.com/DIREKTSPEED-LTD/npm"
npm ERR! node v5.1.1
npm ERR! npm  v3.3.12
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! code EXDEV
npm ERR! errno -18
npm ERR! syscall rename

npm ERR! EXDEV: cross-device link not permitted, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm.DELETE'
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

I'm on aufs.

[IntellixWeb]

Also tried with node and node:slim...no worky :8ball:

[othiym23]

@bcoe, who knows way more about working with Docker than me (as it's what npm On-Site is built upon) has said he'll take a look at this and see if we can't come up with a better way to set things up on the Docker side to avoid this problem.

[shellscape]

@othiym23 @bcoe cheers, much appreciated

[RoryH]

@othiym23 Awesome, solution for this would be great!

[romain-poirson]

Yes indeed !

[frank-dspeed]

ah the problem is your Storage driver i tested your Dockerfile it builds great on my system.

@IntellixWeb

Workaround is you do not do npm install -g http://github* you do git clone http://github* to /usr/local/lib/node_modules/npm and then you cd to that dir and do npm install

then do your normal code

[jgarcows]

I found a work-around, which is to combine my npm calls into a single RUN command. So for the initial Dockerfile, that would be:

RUN npm install -g npm@3; npm install -g npm@3; npm install -g coffee-script

[jvgeee]

@jgarcows did you mean to have npm install -g npm@3 twice?

Also, great timing, was literally just checking this now as I'm coming across the same issues. Will test out your fix and report back.

EDIT: Combining onto one line didn't work for me.

[jgarcows]

@jascination The original problem involved calling it twice, and since I didn't want to post my code as an example, I used his. Sorry to hear it didn't work for you :/

[bcoe]

this problem seems to partially be related to npm's default installation location, one potential workaround is the following:

FROM mhart/alpine-node:5

RUN npm install -g npm --prefix=/usr/local
RUN ln -s -f /usr/local/bin/npm /usr/bin/npm 

CMD [ "/bin/sh" ]

I've tested this approach and was able to circumvent the cross-device link not permitted issue. Not a long term solution perhaps, but I wonder if a fix for this will ultimately be in Docker itself.

[frank-dspeed]

@bcoe your workaround isnt one look it depends simply on the volume use in docker watch the node:5.x images npm works in all on Aufs drivers but when you use volume drivers or mount host volumes into node_modules you need a npm version like the fork that uses move and not rename.

the main problem is that npm uses rename to do what they call atomic write and rename is simply.

so the cross-device problem comes if u use docker with volume driver or you mount one dir or more dirs as host volumes if you run npm in a normal docker container without volumes all will always work as it should

[kevinduffey]

This essentially blocks the ability for Mac users with the latest Docker and boot2docker.iso from being able to use Docker containers with newer versions of npm as that iso is AUFS and until they release one with a different file system, we are stuck, unless there is some way to specify that?

[frank-dspeed]

@kevinduffey please try on your mac because i have none i dont tested it there:

insert into your node image the following

FROM node:5.1

RUN rm -rf /usr/local/lib/node_modules/npm \
 && git clone https://github.com/DIREKTSPEED-LTD/npm /usr/local/lib/node_modules/npm \
 && rm -f  /usr/local/lib/node_modules/npm/.git \
 && rm -f  /usr/local/bin/npm \
 && ln -s -f /usr/local/bin/npm /usr/bin/npm \
 && cd /usr/local/lib/node_modules/npm \
 && npm install

## your normal dockerfile 

[kevinduffey]

I will give that a try.. but I am using a Dockerfile from Ubuntu 14.04.. will the From node:5.1 pull in Ubuntu 14.04?