Increase inotify watch limit

[jankeromnes]

This page shows how to increase the inotify watch limit.

Currently, watching source repositories in a few containers at the same time causes errors like this one (example from a PeerTube container):

> peertube@0.0.15-alpha dev /home/user/PeerTube
> scripty

[1] 
[1] > peertube@0.0.15-alpha watch:server /home/user/PeerTube
[1] > scripty
[1] 
[0] 
[0] > peertube@0.0.15-alpha watch:client /home/user/PeerTube
[0] > scripty
[0] 
[0] 
[0] > peertube-client@0.0.15-alpha ng /home/user/PeerTube/client
[0] > ng "server" "--hmr" "--host" "0.0.0.0" "--port" "3000"
[0] 
[1] [0] 
[1] [0] > peertube@0.0.15-alpha tsc /home/user/PeerTube
[1] [0] > tsc "--sourceMap"
[1] [0] 
[1] [1] 
[1] [1] > peertube@0.0.15-alpha tsc /home/user/PeerTube
[1] [1] > tsc "--sourceMap" "-w"
[1] [1] 
[1] [1] fs.js:1376
[1] [1]     throw error;
[1] [1]     ^
[1] [1] 
[1] [1] Error: watch /home/user/PeerTube ENOSPC
[1] [1]     at _errnoException (util.js:1022:11)
[1] [1]     at FSWatcher.start (fs.js:1374:19)
[1] [1]     at Object.fs.watch (fs.js:1400:11)
[1] [1]     at watchPresentDirectory (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:2635:42)
[1] [1]     at fsWatchDirectory (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:2620:21)
[1] [1]     at Object.watchDirectory (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:2768:28)
[1] [1]     at addDirectoryWatcher (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:56930:21)
[1] [1]     at Object.watchDirectory (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:59955:20)
[1] [1]     at createDirectoryWatcher (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:59381:35)
[1] [1]     at watchFailedLookupLocationOfResolution (/home/user/PeerTube/node_modules/typescript/lib/tsc.js:59348:81)
[1] [1] npm ERR! code ELIFECYCLE
[1] [1] npm ERR! errno 1
[1] [1] npm ERR! peertube@0.0.15-alpha tsc: `tsc "--sourceMap" "-w"`
[1] [1] npm ERR! Exit status 1
[1] [1] npm ERR! 
[1] npm ERR! Failed at the peertube@0.0.15-alpha tsc script.
[1] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[1] [1] 
[1] [1] npm ERR! A complete log of this run can be found in:
[1] [1] npm ERR!     /home/user/.npm/_logs/2018-01-25T10_51_56_946Z-debug.log
[1] [1] npm run tsc -- --sourceMap -w exited with code 1
[1] --> Sending SIGTERM to other processes..
[1] [0] npm run tsc -- --sourceMap && npm run nodemon -- --delay 2 --watch ./dist dist/server exited with code null
[1] npm ERR! code ELIFECYCLE
[1] npm ERR! errno 1
[1] npm ERR! peertube@0.0.15-alpha watch:server: `scripty`
[1] npm ERR! Exit status 1
[1] npm ERR! 
[1] npm ERR! Failed at the peertube@0.0.15-alpha watch:server script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[1] 
[1] npm ERR! A complete log of this run can be found in:
[1] npm ERR!     /home/user/.npm/_logs/2018-01-25T10_51_57_076Z-debug.log
[1] npm run watch:server exited with code 1
--> Sending SIGTERM to other processes..
[0] npm run watch:client exited with code null
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! peertube@0.0.15-alpha dev: `scripty`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the peertube@0.0.15-alpha dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2018-01-25T10_51_57_234Z-debug.log

Process exited with code: 1

Note: Both Kresus and PeerTube projects are known to use the same watch method.

[jankeromnes]

See also https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers

[jankeromnes]

To fix this, we should add the following command to our Makefile:

echo fs.inotify.max_user_watches=99999999 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Note: According to stackoverflow this should allow all users on a given host to collectively use up about 100GB RAM with file watches (i.e. no limit).

EDIT: In the future, it would be nice to use an automated host configuration manager. And also automatically grow and shrink our cluster.

[ishitatsuyuki]

In the case of PeerTube, the Angular CLI can be patched to significantly reduce watch load:

--- client/node_modules/@angular/cli/tasks/serve.js.orig    2018-02-28 16:55:58.239878586 +0900
+++ client/node_modules/@angular/cli/tasks/serve.js 2018-02-28 16:56:28.774147174 +0900
@@ -207,7 +207,8 @@
             proxy: proxyConfig,
             compress: serveTaskOptions.target === 'production',
             watchOptions: {
-                poll: serveTaskOptions.poll
+                poll: serveTaskOptions.poll,
+                ignored: /node_modules/
             },
             https: serveTaskOptions.ssl,
             overlay: {

However, I've had a extremely poor experience contributing to Angular CLI, thus I think it's unlikely this patch would be upstreamed.