fs.watch: node[52551] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

[shama]

I'm getting an error which appears to be directly from libuv while using fs.watch. I'm on OSX 10.8.3 and using node 0.10.5. My ulimit -n is 10480.

I'm able to duplicate the error with the following example (it is a bit contrived but it simulates what is happening with our gruntjs/grunt-contrib-watch module). It only happens when you're watching a larger number of files and another process tries to watch the same files.

I've setup an example in a repo for duplicating the error:

git clone git://github.com/shama/fs-watch-error && cd fs-watch-error
npm install
npm test

---

Here is the code that duplicates the issue:

var fs = require('fs');
var glob = require('glob');
var limit = 9999;
glob('**/*', function(err, files) {
  if (files.length > limit) files = files.slice(0, limit);
  console.log('watching ' + files.length + ' files...');
  files.forEach(function(file) {
    fs.watch(file, function() {});
  });
});

Then run node index.js & node index.js on a large number of files and you should get:

2013-05-13 11:39 node[54911] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

---

I'm looking into updating our module to avoid this error but I'm reporting it here because I believe it shouldn't error out to the console in that way. Considering that I don't seem to be able to catch that error and handle it more appropriately.

Thanks!

[bnoordhuis]

Sorry, can't reproduce. FWIW, it doesn't look like a libuv error to me but rather it's Carbon complaining about failing internal bookkeeping.

/cc @indutny

[shama]

Thanks for trying @bnoordhuis. This is a bit over my head which is why it's an issue not a PR but I'll try looking into it more soon. Good excuse for me to learn. :)

[kevindente]

I'm seeing this problem as well. Only seems to happen with Node 0.10 - I can't reproduce with 0.8.

[bnoordhuis]

v0.8 doesn't use FSEvents.

FWIW, I still think this is an environmental issue rather than a node.js bug but I'll leave the report open for now.

[pho3nixf1re]

I'm getting this same issue on OSX 10.8 with node 0.10.5 using grunt-watch-contrib to watch a large number of files.

[carlos8f]

:+1:

[jamorton]

Same issue as @pho3nixf1re. I also get "Fatal error: EMFILE: Too many opened files." if I keep the watcher open for several hours.

[shama]

@Jonanin EMFILE is a whole different issue. The default max file descriptor limit for OSX is low 256. You can increase it with temporarily with ulimit -n 1200.

FWIW, the easiest way to avoid this error is watch less files because it only happens (for me at least) when watching a large amount of files. Tighten up your globbing patterns. If you're doing **/* then you're probably watching files in your node_modules folder which isn't necessary. Either structure your app different or add '!**/node_modules/**' to your patterns.

[medikoo]

On OSX I see it in WebStorm console (when I run command that uses a lot of watchers), but I don't see it when I run same command in OSX terminal.

[indutny]

Can you please try running disk repair from Disk Utility app? Let me know if this issue is still reproducible after the repair.

[shama]

Still reproducible for me on my MBP:

Also on my older macbook:

Thank you for looking into this @indutny!

[indutny]

I've debugged it up to mach_msg() call which receives some custom-struct which contains that -21 error code.

[indutny]

Apparently, you can watch only 451 files at the same time using FSEvents API. Not sure if its a joke from Apple, or something else... Will try to figure out more soon.

[indutny]

This libuv update should fix it https://github.com/joyent/libuv/pull/894.

[shama]

Awesome! Thank you @indutny!

[indutny]

Landed in libuv's master: https://github.com/joyent/libuv/commit/cd2794c01fc84a4118f79e31071cb1bca78918f5 .

Backporting it to v0.10 before merging into node v0.10: https://github.com/joyent/libuv/pull/896

[thebigredgeek]

Is this fixed?

[alexgorbatchev]

Seeing this on 0.10.25

2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

Darwin alex.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64

[indutny]

It is fixed in v0.11

[ngConsulti]

Any plans to backport this fix into v0.10?

[indutny]

@ngConsulti unfortunatelly no, it is pretty risky and has bitten us a couple of times previously. On a bright side, v0.12 should be quite soon :)

[apaleslimghost]

How soon is "quite soon"? Is there a workaround for 0.10?

[indutny]

There is only one workaround - you could watch less directories.

[indutny]

@tjfontaine should we reconsider backporting?

[apaleslimghost]

There is only one workaround - you could watch less directories.

ಠ_ಠ

Seems to be working fine after a reboot.

[tjfontaine]

I don't think we can backport this to v0.10

[andyburke]

This is affecting me, as well. Would be nice to have it backported to v0.10

[joscha]

Is there any news on this? 0.10.26 on OSX also has this problem with gulp and many watchers.

[chicoxyzzy]

same on 0.10.28

[chicoxyzzy]

@tjfontaine @indutny I think fix should be backported to 0.10 cause of many users of 0.10 versions. Not everyone will start using 0.12 immediately. More then that 0.12 is not released yet. I can try to backport the fix. Or is it really so difficult?

[indutny]

@chicoxyzzy sorry, I don't have time for it, considering the soonish v0.12 release.

If you are really interested in working it out - here are the relevant libuv commits that are fixing it:

cd2794c01fc84a4118f79e31071cb1bca78918f5 82f2472b9f51b1bbb17873ae0d00fce945d5e294 429bb804ed26e1928818907b1cfcf0161c954ea8 0fdd99f02b2b22b0ba4e23f70017fa77393caa11 43bef41031791b16ca94a58c33d6464e3e543e40 b1b931de4e634be515e528d1d07d59551a86eded 0f5c28b6844dbe8c037aadfa291c9e6277b2d0e9 bf5038df4008d894409dac4d6e64a94f5630bf61 646de34f5e83994f121892394776c60cb01a2706

[macoii25a]

Hi have the same problem by running grunt server, do you know what could be happening?

[indutny]

The grunt is trying to watch a lot of directories at once (usually about 400), and that's where odd things are starting to happen with OSX. We have fixed it in node.js v0.11, considering that v0.12 will be soon out, I'd recommend you to give it a try and see if it does work with your code ;)

[shama]

The grunt is trying to watch a lot of directories at once (usually about 400)

Grunt only watches whatever the user has told it to watch. The most common reason users hit this issue is because they are accidentally watching their entire node_modules folder with a pattern like: **/*.js.

@macoii25a Be sure you're not accidentally watching the node_modules folder. You can view all the files grunt watch is watching by running grunt watch -v. If so, exclude the node_modules folder by adding '!**/node_modules/**' or moving to a more easily targeted folder, such as src/**/*.js.

Thanks again @indutny for this fix. Looking forward to node v0.12!

[macoii25a]

I've tried to excluding node_modules but it's also throwing the error. Being more specific, the error happens after trying to compile less files. Also I have installed node v0.10.28 by brew

[shama]

@macoii25a Could you open an issue on https://github.com/gruntjs/grunt-contrib-less with details about the error and which OS? Thanks!

[chicoxyzzy]

@indutny all of those links are 404

[indutny]

They are libuv commit hashes:

cd2794c01fc84a4118f79e31071cb1bca78918f5
82f2472b9f51b1bbb17873ae0d00fce945d5e294
429bb804ed26e1928818907b1cfcf0161c954ea8
0fdd99f02b2b22b0ba4e23f70017fa77393caa11
43bef41031791b16ca94a58c33d6464e3e543e40
b1b931de4e634be515e528d1d07d59551a86eded
0f5c28b6844dbe8c037aadfa291c9e6277b2d0e9
bf5038df4008d894409dac4d6e64a94f5630bf61
646de34f5e83994f121892394776c60cb01a2706

[apaleslimghost]

lftl:

https://github.com/joyent/libuv/commit/cd2794c01fc84a4118f79e31071cb1bca78918f5 https://github.com/joyent/libuv/commit/82f2472b9f51b1bbb17873ae0d00fce945d5e294 https://github.com/joyent/libuv/commit/429bb804ed26e1928818907b1cfcf0161c954ea8 https://github.com/joyent/libuv/commit/0fdd99f02b2b22b0ba4e23f70017fa77393caa11 https://github.com/joyent/libuv/commit/43bef41031791b16ca94a58c33d6464e3e543e40 https://github.com/joyent/libuv/commit/b1b931de4e634be515e528d1d07d59551a86eded https://github.com/joyent/libuv/commit/0f5c28b6844dbe8c037aadfa291c9e6277b2d0e9 https://github.com/joyent/libuv/commit/bf5038df4008d894409dac4d6e64a94f5630bf61 https://github.com/joyent/libuv/commit/646de34f5e83994f121892394776c60cb01a2706

[jnovack]

I just want to +1 the following from @shama.

Be sure you're not accidentally watching the node_modules folder. You can view all the files grunt watch is watching by running grunt watch -v. If so, exclude the node_modules folder by adding '!**/node_modules/**' or moving to a more easily targeted folder, such as 'src/**/*.js'.

[pspi]

Seeing this with node v0.10.30

2014-08-18 17:55 grunt[11156] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

[apaleslimghost]

@pspi yeah, it's not gonna get fixed in 0.10.x.

[yelvert]

Then when is 0.12 gonna be release, this is getting ridiculous, every person using watch on OSX has this problem. You've been saying 0.12 would be release soon since May, its September, soon was months ago.

[Grawl]

@yelvert I am the one who do not have this problem. I am using a narrow file lists for watching.

[mstoltenburg]

I am using a narrow file too (or is 23 files considered huge), but it seems like I'm having the same problem. Are there any news?

[Grawl]

@mstoltenburg it's strange. Try to update something. I am watching for all my Bower installed packages + all my sources (about 50 files per project) and it working good.

[mstoltenburg]

@Grawl I updated constantly yesterday to get rid of this error - nothing happend. Today the error disappeared, maybe the reboot did the trick. I'm sorry I can't tell better what the triggering trouble child was.

[eedeebee]

experiencing this joy in 0.10.32 myself today

[ognjenm]

"experiencing this joy in 0.10.32 myself today" same here

[SimonKaluza]

I have this issue as well. Looking forward to version 0.12.

EDIT

Installing Node version 0.11.14 cleared up the issue for me.