Use a filesystem change listener rather than polling for project changes

[jlaxson]

The sc-server thread that watches the project directory for changes to rebuild polls every two seconds, and for me consumes up to around 10% of a CPU just sitting there. This pull implements the 'listen' gem, which uses inotify, fsevent, or other available facility to be notified of changes. This drops idle CPU usage of sc-server to almost (but not quite) nothing.

[publickeating]

Brilliant!

[mauritslamers]

Just a question: how does this "listen" library deal with the problem of the insanely small amount of open files allowed under MacOSX? One of the reasons I haven't used this extensively in the nodejs build tools is that MacOSX only allows 256 files open simultaneously. This requires the user to set the amount of files much higher every time the build tools are started (ulimit -n).

EDIT: 256 instead of 250

[jlaxson]

On OSX, it uses the fsevent facility to subscribe to notifications for the project directory. This is a system-level service and communicates (I presume) with mach ports. So the listener doesn't need to touch the filesystem at all, much less have every file open at once.

For the purposes of Abbot I did leave in the current code that timestamps everything, for fear of making huge changes all at once, but it can probably be trashed once we're sure the listener is working well.

[mauritslamers]

That is indeed a different strategy. NodeJS uses kqueue which will by default seems to take one file descriptor per watcher.

[dcporter]

This conversation belongs on the build-tools repo somewhere, but: node is going to very much want to learn how to use the fsevent APIs when on a mac. (Those aren't exactly new APIs.) Would it make sense to apply some pressure via github issues etc. somewhere in node's stack?

[mauritslamers]

It is just that the default fs.watch method does not use it. There are ways of using fsevents when on OSX, such as https://npmjs.org/package/fsevents.

[publickeating]

Is it just me or does this change use fs change events to then manually go through and check for changes? I've noticed that my Ruby process is pushing 100% and after some blind Ruby debugging, I think it probably has something to do with the block taking too long to complete before another fs change event comes in (this is a bit of a wild guess).

In any case, I think that if we have fs change events, then we can skip adding up all the files and their modified times and simply note that the project did change.

[publickeating]

I believe I've fixed up the 100% CPU problem and improved the performance the way @jlaxson intended. Please verify on master Abbot.

Here's the commit: https://github.com/sproutcore/abbot/commit/7052d243b4f196f5d2a4901a868a5baedfddae1d