`make watch` task taking ~75% CPU

[tremby]

When I run make watch on tower it does all its building and then settles into the watch state, during which it sits at ~75% CPU. Any idea what might be causing this?

I'm running Ubuntu. Everything (except maybe some symlinks/bootstrap scripts put down by sudo npm link) is on the same partition.

[edubkendo]

Did you see my screenshot ? Was I reading that correctly? I believe I'm getting a drastically different result: https://dl.dropbox.com/u/24245334/grunt_htop.png

[tremby]

Different result? Looks like the same result to me -- grunt is taking up 70% of one of your cores. That's just madness for watching for file system changes. Must be something funky happening.

[edubkendo]

Ah ok, I wasn't seeing that line lol. I was seeing the line lower down, where it said grunt was using 1%. And completely missed the one higher up until just now. Just literally didn't see it.

[lancejpollard]

If you want to try making a simple node wrapper to this ruby gem, the watcher will become seriously optimized:

https://github.com/guard/listen

I was getting <2% CPU after initialization.

Maybe we can say, "if you want better performance with the watcher, then gem install listen if you have ruby installed." Then we just check if we can spawn a simple ruby script that just streams the output from the listen gem. This is all it really takes:

• run the ruby script from node.js: https://github.com/viatropos/design.io/blob/master/src/design.io/listener/mac.coffee#L7-L22
• and the ruby script looks like this: https://github.com/viatropos/design.io/blob/master/lib/design.io/listener/mac.rb

So for the listen gem, the ruby script would probably just be like:

require 'rubygems'
require 'listen'
require 'json'

STDOUT.sync = true
io          = STDOUT
directory   = ARGV[0] || Dir.pwd
latency     = ARGV[1] || 0.2

listener = Listen.to(directory)
listener.latency(latency)
listener.change do |modified, added, removed|
  # just parse this json stream in node.
  io.write(JSON.stringify('modified' => modified, 'added' => added, 'removed' => removed))
end
listener.start

If anyone's interested in getting this going let me know.

[lancejpollard]

What would be even more awesome is if we wrapped this in a top-level API for Tower, so you could watch files in an optimized way anywhere you want, such as:

Tower.watch process.cwd(), (files) ->
  files.changed # ['package.json']
  files.added # []
  files.removed # []

where Tower.watch would try to use this custom listen wrapper if possible, otherwise falling back to fs.watch (or better, gaze).

[lancejpollard]

@edubkendo and @tremby if you gem install listen, which uses rb-inotify (https://github.com/guard/listen/blob/master/lib/listen/adapters/linux.rb), and run the following to watch the entire directory tree, is it still using the same (large) amount of CPU?

# file named watch.rb
require 'rubygems'
require 'listen'

listener = Listen.to(Dir.pwd)
listener = listener.latency(0.2)
listener = listener.change do |modified, added, removed|
  puts modified, added, removed
end
listener.start

If you add that to the root of your tower repo, name it watch.rb, and run:

ruby watch.rb

and then change some nested files, you should see the names of the changed files in the terminal.

When you do this, what does your CPU usage look like?

My CPU, on a MacBook Pro upgraded to 10.8.2 with Xcode 4.5.2 installed, is only at <1%. The best part is, I can change any file in the directory tree and will get notified; it doesn't seem to matter how many files there are. Though, this listen gem is computing a fingerprint from the file if the mtime hasn't changed, which can be kind of slow on very large files (more than a few megabytes, which isn't relevant when it comes to watching source code files, only PDFs, data-dumps, images, etc.).

On a Mac this watch method is WAY better than what Node.js does. Node's watcher on the Mac uses kqueue, which gives you file-level notifications. The alternative is FSEvent which gives you directory-level notifications, on entire trees of files. For kqueue, you have to watch each individual file, for FSEvent, only a single directory.

On Linux though, Node.js uses inotify, and this listen rubygem uses it also. So there may be no benefit there on Linux, but I'm interested to see if you guys see any performance improvement. If so, I'll make a watcher module for Node that wraps this gem, otherwise I'll just do something simpler for Mac-only.

Some relevant node.js issues on their watcher:

• https://github.com/joyent/node/issues/1986
• https://github.com/joyent/node/issues/2023
• https://github.com/joyent/node/issues/2062
• https://github.com/thibaudgg/rb-fsevent/issues/38

[thehydroimpulse]

@viatropos Haven't tried this on Linux, but using that ruby watcher runs at around 10Mb of RAM, compared to node (watchr) at 21Mb. Seems like watchr (https://github.com/bevry/watchr) uses less memory then grunt's watcher or chokidar, on Windows at least.

[lancejpollard]

@TheHydroImpulse good to know. So if you find out more about which gem/module is the most optimized for the different operating systems (windows, linux, mac), definitely post a link. Over time perhaps we can compile a list of the best/fastest watchers for each OS and wrap them up for node.

• https://github.com/thibaudgg/rb-fsevent is the fastest on a Mac. We could wrap the raw C binary @ttilley created (the fsevent_watch command https://github.com/thibaudgg/rb-fsevent/issues/38#issuecomment-10925351).
• https://github.com/guard/listen wraps rb-fsevent, and has linux/windows adapters, but I only know rb-fsevent is the best for Mac. This listen gem also has a nice bit of code wrapping rb-fsevent so it gives you the exact file that changed, whereas FSEvent only tells you of the directory.

[thehydroimpulse]

Not sure why Node has so much problem with file watchers.

You could create a rb-fsevent "clone" in C or C++ that would become a Node module or part-module for the mac part. FSEvent is a fairly straight forward system: https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html

Maybe we should create a Tower.watch wrapper around many different other solutions.

• fsevent or rb-fsevent are the fastest (Mac)
• kqueue uses file descriptors for identification of monitored files - Meaning it has to open each file it watches.
• inotify is the only linux solution, though dnotify exists, inotify replaces it as it's much better. EDIT: fschange would be fastest.
• Windows only has it's one event: ReadDirectoryChangesW

Also inotify cannot recursively watch sub-directories. You have to create a new watcher for each of them. "dnotify and inotify are the current file change notification services in the Linux kernel. Unfortunately, they share a couple of shortcomings that make it difficult to use them for efficient (real-time) file system indexing under Linux. One of them is their unability to detect content-preserving file changes, i.e. write operations that only affect a tiny portion of a file. It is impossible to use inotify in order to determine what part of a file has actually been modified. A second problem is that you have to open every directory once before you can register for changes within that directory. This is very unconvenient, especially if you want to watch large file systems, as it requires a full scan of the file system." - http://stefan.buettcher.org/cs/fschange/index.html

It appears that linux has another solution called fschange that is much more efficient. The problem is that you need to patch your kernel or replace some files within the kernel to use it. A solution would be to use their fschange daemon: http://stefan.buettcher.org/cs/fschange/inotifyd-2005-11-19.tgz that emulates fschange using inotify.

This might solve the problem for all platforms. Though you'd need to create some C/C++ node modules, but that's not really a big deal.

[lancejpollard]

Yeah that sounds good. Ideally there is a general module (like https://github.com/shama/gaze, which is a good start for node) that anyone can use for better, intuitive, fast file watching support in node. Then we wrap that in Tower so you can do:

Tower.watch('./**/*.js', function(files) {
  files.changed
  files.added
  files.removed
});

The API could also be similar to guard/listen:

Tower.watch('./**/*.js')
  .ignore(/node_modules/)
  .filter(/Test\.js$/)
  .latency(0.2)
  .forcePolling(true)
  .pollingFallbackMessage(false)
  .on('changed', function(files) { })
  .on('removed', function(files) { })
  .on('added', function(files) { })
  .start();

...where essentially Tower.watch() is just returning something like a stream.

I'll be messing with rb-fsevent and the executable (https://github.com/thibaudgg/rb-fsevent/issues/38#issuecomment-10925351) to speed up node development for Tower - and to stop killing my fan, which now has to be replaced :p

[ttilley]

I know very little about creating proper node.js extension modules, but the code in fsevent_watch is fairly straightforward and takes care of frustrating issues such as building on 10.5 while still using features from 10.7 when running there. This just involves a small amount of compatibility code and a gestalt check (to see what version of OSX you're currently running on). It's unfortunate that compile-time and run-time checks need to both be performed, but I wanted a build of fsevent_watch to just work and run anywhere. I'd assume you would want the same for a compiled extension.

...hell, it might be fun to take the time to learn the API for extending node.

[thehydroimpulse]

@viatropos If I remember correctly gaze doesn't work on Windows, or partially work. deleted and added events don't fire. But other then that, it's a really good watcher. Love the minimatch/glob support.

[shama]

@TheHydroImpulse gaze should fully work on Windows, if not please open an issue :)

[thehydroimpulse]

@viatropos @shama ahh yes, gaze does indeed fully work on Windows.