Operation not supported error when using NFS mount

[chadrik]

I get an error due to the use of flock in the ExecFlock function within gyp/pylib/gyp/mac_tool.py

chad$ npm install contextify
\
> contextify@0.1.11 install /path/to/shared/data/node_modules/contextify
> node-gyp rebuild

  CXX(target) Release/obj.target/contextify/src/contextify.o
  SOLINK_MODULE(target) Release/contextify.node
Traceback (most recent call last):
  File "./gyp-mac-tool", line 512, in <module>
    sys.exit(main(sys.argv[1:]))
  File "./gyp-mac-tool", line 28, in main
    exit_code = executor.Dispatch(args)
  File "./gyp-mac-tool", line 43, in Dispatch
    return getattr(self, method)(*args[1:])
  File "./gyp-mac-tool", line 216, in ExecFlock
    fcntl.flock(fd, fcntl.LOCK_EX)
IOError: [Errno 45] Operation not supported
make: *** [Release/contextify.node] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:810:12)
gyp ERR! System Darwin 14.0.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /path/to/shared/data/node_modules/contextify
gyp ERR! node -v v0.10.33
gyp ERR! node-gyp -v v1.0.2
gyp ERR! not ok 
npm ERR! Darwin 14.0.0
npm ERR! argv "node" "/usr/local/bin/npm" "install" "contextify"
npm ERR! node v0.10.33
npm ERR! npm  v2.1.6
npm ERR! code ELIFECYCLE

I replaced ExecFlock with this recipe and it resolved the error. Unfortunately, the code is lgpl.v2.

[wesleytodd]

Possibly related. When we run npm install on a directory mounted with nfs we get this error:

make: Entering directory `/usr/local/src/v/node/node_modules/node-xmpp/node_modules/node-expat/node_modules/iconv/build'
  CC(target) Release/obj.target/libiconv/deps/libiconv/lib/iconv.o
  AR(target) Release/obj.target/iconv.a
  COPY Release/iconv.a
cp: preserving permissions for `Release/iconv.a': Operation not permitted
make: *** [Release/iconv.a] Error 1

node-gyp is doing a cp -af that is not supported over NFS.

[aklinkert]

Any possibilities to get this fixed @TooTallNate? Same issue over here:

make: Warning: File `hashvalue.target.mk' has modification time 36 s in the future
  CXX(target) Release/obj.target/hashvalue/src/hashvalue.o
  SOLINK_MODULE(target) Release/obj.target/hashvalue.node
  COPY Release/hashvalue.node
cp: preserving permissions for ‘Release/hashvalue.node’: Operation not permitted
make: *** [Release/hashvalue.node] Error 1

[pkyeck]

node-gyp is doing a cp -af that is not supported over NFS.

any way around this? running into same "problems".

[aronwoost]

+1

[chiefy]

:+1: Just ran into this trying to npm rebuild on a NFS mounted nodejs project inside a Vagrant env. :cry:

[bnoordhuis]

If you get that flock error, just run env LINK=g++ npm install whatever to skip the gyp-mac-tool script.

There is no way around that cp -af command though. If you get an EPERM error, then maybe there's a configuration mismatch between the server and the client, like user ids not mapping 1-to-1?

[aleksanyan]

+1

[pkyeck]

for all you vagrant users out there - I used https://github.com/gael-ian/vagrant-bindfs to fix the errors when trying to npm install on a NFS mount

[aleksanyan]

+1

[ifraixedes]

@pkyeck Thanks, it fixed perfectly the problem

[armchairdj]

+1 Thanks @pkyeck!

Vagrantfile:

config.vm.synced_folder '.', '/vagrant', nfs: { mount_options: ['actimeo=2'] }
# Fix file permissions issues that will kill npm install on the Vagrant box.
config.bindfs.bind_folder '/vagrant', '/vagrant'

Host:

vagrant plugin install vagrant-bindfs
vagrant up

No more node-gyp errors when doing npm install on modules with native extensions. I spent 2 full days convinced the problem was with Ubuntu 14.04 LTS, libxmljs or node-gyp. It was just vagrant and NFS!

[awreece]

I don't get the flock issue, but I do get the cp issue:

... <snip> ...
  rm -rf "Release/sass.a" && cp -af "Release/obj.target/src/sass.a" "Release/sass.a"
cp: preserving permissions for ‘Release/sass.a’: Operation not permitted
make[1]: *** [Release/sass.a] Error 1
make[1]: Leaving directory `/platform/node_modules/node-sass/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/platform/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Linux 3.16.0-62-generic
gyp ERR! command "/usr/bin/nodejs" "/platform/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd /platform/node_modules/node-sass
gyp ERR! node -v v5.1.1
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok 
Build failed

The vagrant (or equivalently HGFS) workarounds are cute but insufficient: I've experienced brutal (> 10x) build performance penalties when building on such filesystems. Why is the cp -af necessary; is there a path forward to test not using that flag, trying again without the flag and warning, or using filesystem specific flags? If the issue is "yes, $SOLUTION is the right path forward but we don't have engineer time right now" I can probably help with the testing.

[bnoordhuis]

@awreece Tryenv cmd_copy='ln -f $< $@ || cp -PRfp $< $@' node-gyp rebuild

[awreece]

@bnoordhuis Sorry for the latency on the reply; unfortunately I was unable to run your command successfully. I think I need to run the node-gyp rebuild in the source repo of one of the packages that is failing (rather than from the directory in which I run npm install); however, I'm having some difficulties with that. Is there a way I can tell npm to not destroy intermediate directories so I can test?

I will give you this, I tested both parts of the copy_cmd: the former works on my nfs share and the latter does not:

/platform/test/node-sass-3.4.2 $ cp -PRfp alex alex2
cp: preserving permissions for ‘alex2’: Operation not permitted
/platform/test/node-sass-3.4.2 $ ln -f alex alex2

[bnoordhuis]

Does cp -PRf alex alex2 work (i.e., without the -p switch)? To answer your other question, I think env cmd_copy='...' npm install should work but if it doesn't, try cloning the node-sass repo.

[awreece]

• cp -PRf alex alex2 was successful.
• env cmd_copy='ln -f $< $@ || cp -PRfp $< $@' npm install was unsuccessful, but I'm not convinced the cmd_copy had an effect:

rm -rf "Release/sass.a" && cp -af "Release/obj.target/src/sass.a" "Release/sass.a"
cp: preserving permissions for ‘Release/sass.a’: Operation not permitted
make: *** [Release/sass.a] Error 1

• I tried cloning node-sass and building it, but I'm getting (presumably) unrelated build errors:

gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/platform/test/node-sass-3.4.2/build'
make: *** No rule to make target `Release/obj.target/libsass/src/libsass/src/ast.o', needed by `Release/obj.target/src/sass.a'.  Stop.
make: Leaving directory `/platform/test/node-sass-3.4.2/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/platform/test/node-sass-3.4.2/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Linux 3.16.0-62-generic
gyp ERR! command "/usr/bin/nodejs" "/platform/test/node-sass-3.4.2/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="

I'd prefer not to dig into understanding the node-sass build system; do you think there is a way to get the npm invocation working? Alternatively, I could try a "hello world" node-gyp package (is there such an example in a test directory or something?).

[bnoordhuis]

I'm guessing npm doesn't pass on the environment to child processes.

I could try a "hello world" node-gyp package (is there such an example in a test directory or something?).

Does npm install buffertools (with/without the env cmd_copy='...') work?

[awreece]

For modern versions of nodejs (v5.1.1) and npm (v3.3.12), neither work:

• env cmd_copy='ln -f $< $@ || cp -PRfp $< $@' npm install buffertools
• npm install buffertools

As before, I'm not conviced the cmd_copy is being forwarded:

  COPY Release/buffertools.node
cp: preserving permissions for 'Release/buffertools.node': Operation not permitted
make: *** [Release/buffertools.node] Error 1
make: Leaving directory `/platform/test/node_modules/buffertools/build'

Running execsnoop, I can see it using the old copy command:

 11464  11435 printf %s\n   COPY Release/buffertools.node
 11465  11435 /bin/sh -c mkdir -p "Release/" "./Release/.deps/Release/"
 11466  11465 mkdir -p Release/ ./Release/.deps/Release/
 11467  11435 /bin/sh -c rm -rf "Release/buffertools.node" && cp -af "Release/obj.target/buffertools.node" "Release/buffertools.node"
 11468  11467 rm -rf Release/buffertools.node
 11469  11467 cp -af Release/obj.target/buffertools.node Release/buffertools.node

For kicks, I tried just using npm install with a package.json and (as expected) there was no difference:

{
  "name": "memsql-platform",
  "devDependencies": {
    "buffertools": "2.1.3"
  }
}

Interestingly, I forced the use of an old node (v0.10.25) and npm (1.3.10) and the package installed successfully:

alex@54bb1aa92018:/platform/test$ PATH=/usr/bin:$PATH npm install
npm WARN package.json memsql-platform@ No description
npm WARN package.json memsql-platform@ No repository field.
npm WARN package.json memsql-platform@ No README data
npm http GET https://registry.npmjs.org/buffertools/2.1.3
npm http 304 https://registry.npmjs.org/buffertools/2.1.3

> buffertools@2.1.3 install /platform/test/node_modules/buffertools
> node-gyp rebuild

make: Entering directory `/platform/test/node_modules/buffertools/build'
  CXX(target) Release/obj.target/buffertools/buffertools.o
  SOLINK_MODULE(target) Release/obj.target/buffertools.node
  SOLINK_MODULE(target) Release/obj.target/buffertools.node: Finished
  COPY Release/buffertools.node
make: Leaving directory `/platform/test/node_modules/buffertools/build'
buffertools@2.1.3 node_modules/buffertools
alex@54bb1aa92018:/platform/test$ echo $?
0

With execsnoop, I can confirm that this is using the symlink approach instead of the cp approach:

 11402  11371 printf %s\n   COPY Release/buffertools.node
 11403  11371 /bin/sh -c mkdir -p "Release/" "./Release/.deps/Release/"
 11404  11403 mkdir -p Release/ ./Release/.deps/Release/
 11405  11371 /bin/sh -c ln -f "Release/obj.target/buffertools.node" "Release/buffertools.node" 2>/dev/null || (rm -rf "Release/buffertools.node" && cp -af "Release/obj.target/buffertools.node" "Release/buffertools.node")
 11406  11405 ln -f Release/obj.target/buffertools.node Release/buffertools.node

These versions are the default in the ubuntu distro I'm testing with, but seem ancient:

alex@54bb1aa92018:/platform/test$ PATH=/usr/bin:$PATH node -v           
v0.10.25
alex@54bb1aa92018:/platform/test$ PATH=/usr/bin:$PATH npm -v
1.3.10
alex@54bb1aa92018:/platform/test$ PATH=/usr/bin:$PATH node-gyp -v
v0.10.10

For reference, the modern versions I tested with are:

alex@54bb1aa92018:/platform/test$ node-gyp -v
v3.3.1
alex@54bb1aa92018:/platform/test$ npm -v
3.3.12
alex@54bb1aa92018:/platform/test$ node -v
v5.1.1

[MylesBorins]

hey @awreece they most recent version of the v5 release line is v5.9.1, would you be able to update and see if that fixes anything?

[awreece]

@TheAlphaNerd I tried on node v5.9.1 and got the same failure. For reference, I don't believe the issue is related to node at; the problem appears to be that node-gyp is executing cp -af (which fails on nfs mounts) and should be unrelated to the version of node.

[bnoordhuis]

@awreece Keep in mind you probably need to drop p from cp -PRfp in order for it to work. Try passing V=1 to env as well (besides cmd_copy, that is.)

[MylesBorins]

@awreece I've removed my response... thanks for updating your comment

[awreece]

@bnoordhuis D'oh, you're right that I forgot to drop the -p. It fails in the same way:

$ env V=1 cmd_copy='ln -f $< $@ || cp -PRf $< $@' npm install buffertools

> buffertools@2.1.3 install /platform/test/node_modules/buffertools
> node-gyp rebuild

make: Entering directory `/platform/test/node_modules/buffertools/build'
  g++ '-DNODE_GYP_MODULE_NAME=buffertools' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DBUILDING_NODE_EXTENSION' -I/home/alex/.node-gyp/5.9.1/include/node -I/home/alex/.node-gyp/5.9.1/src -I/home/alex/.node-gyp/5.9.1/deps/uv/include -I/home/alex/.node-gyp/5.9.1/deps/v8/include  -fPIC -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -ffunction-sections -fdata-sections -fno-omit-frame-pointer -fno-rtti -fno-exceptions -std=gnu++0x -MMD -MF ./Release/.deps/Release/obj.target/buffertools/buffertools.o.d.raw   -c -o Release/obj.target/buffertools/buffertools.o ../buffertools.cc
  g++ -shared -pthread -rdynamic -m64  -Wl,-soname=buffertools.node -o Release/obj.target/buffertools.node -Wl,--start-group Release/obj.target/buffertools/buffertools.o -Wl,--end-group 
  rm -rf "Release/buffertools.node" && cp -af "Release/obj.target/buffertools.node" "Release/buffertools.node"
cp: preserving permissions for 'Release/buffertools.node': Operation not permitted
make: *** [Release/buffertools.node] Error 1
make: Leaving directory `/platform/test/node_modules/buffertools/build'

Between this additional information and the execsnoop output from earlier, it really doesn't seem like the cmd_copy is having an effect.

[bnoordhuis]

Okay, I think the conclusion has to be that npm doesn't pass on the environment to child processes. There's probably a switch for it but I don't know what it is. Maybe try filing an issue over at https://github.com/npm/npm/issues?

[awreece]

When I run node-gyp directly in the node-buffertools repo, I get the same behavior though (that is, that cmd_copy has no effect):

alex@dcdb3872bc2c:/platform/test/node-buffertools$ env V=1 cmd_copy='ln -f $< $@ || cp -PRf $< $@' /usr/local/bin/node-gyp rebuild
gyp info it worked if it ends with ok
gyp info using node-gyp@3.3.1
gyp info using node@5.9.1 | linux | x64
gyp info spawn /usr/bin/python2
... <snip> ...
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/platform/test/node-buffertools/build'
  g++ '-DNODE_GYP_MODULE_NAME=buffertools' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DBUILDING_NODE_EXTENSION' -I/home/alex/.node-gyp/5.9.1/include/node -I/home/alex/.node-gyp/5.9.1/src -I/home/alex/.node-gyp/5.9.1/deps/uv/include -I/home/alex/.node-gyp/5.9.1/deps/v8/include  -fPIC -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -ffunction-sections -fdata-sections -fno-omit-frame-pointer -fno-rtti -fno-exceptions -std=gnu++0x -MMD -MF ./Release/.deps/Release/obj.target/buffertools/buffertools.o.d.raw   -c -o Release/obj.target/buffertools/buffertools.o ../buffertools.cc
  g++ -shared -pthread -rdynamic -m64  -Wl,-soname=buffertools.node -o Release/obj.target/buffertools.node -Wl,--start-group Release/obj.target/buffertools/buffertools.o -Wl,--end-group 
  rm -rf "Release/buffertools.node" && cp -af "Release/obj.target/buffertools.node" "Release/buffertools.node"
cp: preserving permissions for 'Release/buffertools.node': Operation not permitted
make: *** [Release/buffertools.node] Error 1
make: Leaving directory `/platform/test/node-buffertools/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:100:13)
gyp ERR! stack     at ChildProcess.emit (events.js:185:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Linux 3.16.0-62-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/bin/node-gyp" "rebuild"
gyp ERR! cwd /platform/test/node-buffertools
gyp ERR! node -v v5.9.1
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok 

[awreece]

I manually patched the makefile generator to respect the value from the environment for cmd_copy:

$ diff -u /usr/local/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py{.bak,}
--- /usr/local/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py.bak    2016-03-31 19:07:44.650070166 +0000
+++ /usr/local/lib/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py    2016-03-31 19:07:56.133414301 +0000
@@ -365,7 +365,7 @@

 quiet_cmd_copy = COPY $@
 # send stderr to /dev/null to ignore messages when linking directories.
-cmd_copy = rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@"
+cmd_copy ?= rm -rf "$@" && cp %(copy_archive_args)s "$<" "$@"

 %(link_commands)s
 """

Re ran env V=1 cmd_copy='ln -f $< $@ || cp -PRf $< $@' /usr/local/bin/node-gyp rebuild, and it was successful!

alex@dcdb3872bc2c:/platform/test/node-buffertools$ env V=1 cmd_copy='ln -f $< $@ || cp -PRf $< $@' /usr/local/bin/node-gyp rebuild
gyp info it worked if it ends with ok
gyp info using node-gyp@3.3.1
gyp info using node@5.9.1 | linux | x64
gyp info spawn /usr/bin/python2
... <snip> ...
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/platform/test/node-buffertools/build'
  g++ '-DNODE_GYP_MODULE_NAME=buffertools' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DBUILDING_NODE_EXTENSION' -I/home/alex/.node-gyp/5.9.1/include/node -I/home/alex/.node-gyp/5.9.1/src -I/home/alex/.node-gyp/5.9.1/deps/uv/include -I/home/alex/.node-gyp/5.9.1/deps/v8/include  -fPIC -pthread -Wall -Wextra -Wno-unused-parameter -m64 -O3 -ffunction-sections -fdata-sections -fno-omit-frame-pointer -fno-rtti -fno-exceptions -std=gnu++0x -MMD -MF ./Release/.deps/Release/obj.target/buffertools/buffertools.o.d.raw   -c -o Release/obj.target/buffertools/buffertools.o ../buffertools.cc
  g++ -shared -pthread -rdynamic -m64  -Wl,-soname=buffertools.node -o Release/obj.target/buffertools.node -Wl,--start-group Release/obj.target/buffertools/buffertools.o -Wl,--end-group 
  ln -f Release/obj.target/buffertools.node Release/buffertools.node || cp -PRf Release/obj.target/buffertools.node Release/buffertools.node
make: Leaving directory `/platform/test/node-buffertools/build'
gyp info ok 

[pasupulaphani]

@pkyeck Thank u. bindfs fixed my issue.

[awreece]

Any update on this?

[awreece]

Any update on this? It seems the current state of the ticket:

• the problem appears to be that node-gyp is executing cp -af (which fails on nfs mounts)
• switching cmd_copy to ln -f $< $@ || cp -PRf $< $@ resolved the issue in at least one environment

Is there additional testing that needs to happen?

[crabmusket]

For others dealing with packages that have this issue (I ran into it when trying to install the canvas package), I found that installing the package globally then linking it into my local (shared) project worked:

sudo npm install -g canvas
npm link canvas

I believe this works because installing the package globally puts it in a non-shared location where this issue doesn't occur. Note that npm link uses a symlink so you may have issues with e.g. Linux guests on Windows hosts (I can never quite get that to work properly).

[jleroy]

You can mount your NFS share with the noacl option in order to avoid this error.

Example (/etc/fstab): XXX.XXX.XXX.XXX:/remote/path /local/path nfs _netdev,auto,noacl 0 0

[jimklimov]

@jleroy : thanks for the hint, helped me out (in a different project, but similar problem) :) :+1:

[squidfunk]

+1

[smurfy]

+1

[rvagg]

Probably needs to be resolved in https://github.com/refack/GYP rather than here, @refack do you agree? Can we close this with the suggestion that someone is welcome to open a PR there and have it downstreamed here eventually?

[mrigor]

For the Vagrant folks, this is another solution: yarn install --modules-folder /tmp/node_modules && rsync -a --no-owner --no-group /tmp/node_modules /vagrant/ (build outside of NFS mount and move it in, ignoring permissions)

[bnoordhuis]

To summarize:

• node-gyp currently uses cp -af
• cp -af is functionally identical to cp -pPRf
• -p causes trouble for some NFS users because it copies ACLs (fix: mount with noacl)
• copying ACLs is usually the desired behavior though

I'm closing this but whoever feels dropping -p is a good idea, feel free to take this patch and PR it, and we'll discuss its ramifications.

diff --git a/gyp/pylib/gyp/generator/make.py b/gyp/pylib/gyp/generator/make.py
index 26cf88c..f40a80a 100644
--- a/gyp/pylib/gyp/generator/make.py
+++ b/gyp/pylib/gyp/generator/make.py
@@ -2044,7 +2044,7 @@ def GenerateOutput(target_list, target_dicts, data, params):
     srcdir_prefix = '$(srcdir)/'

   flock_command= 'flock'
-  copy_archive_arguments = '-af'
+  copy_archive_arguments = '-PRf'
   makedep_arguments = '-MMD'
   header_params = {
       'default_target': default_target,

[mike-potter]

Sad this was closed with no followup or fix.

• Adding "noacl" as mentioned above did not fix it for me.
• Performance issues when using "bindfs" which slows down all other file operations, such as "composer install".
• Ended up using the method above to do the "npm ci" in the /tmp folder and then move the results into the nfs destination. Still a performance impact but at least it only affects npm and I can decide to only do this for sites that have a dependency that needs node-gyp (in our case it is "ttf2woff2")

[H4wKs]

This is not a nodejs issue, but more likely the environment used. I had the issue running vagrant with Parallels on M3, which do not have a noacl mount option. The only work around I found was to run the command with sudo, and then the permission error was gone. Not a super nice way to do it, but that's how I got it fixed.