Open vovimayhem opened 8 years ago
Things like 'bower_components' and 'node_modules' should be ignored by Listen default. (I would've done that a long time ago, except I was worried about SemVer - I was probably too careful).
Listen has a default list of files/directories to ignore. I think both should be added there. I doubt anyone would get hurt if I just do a minor version bump.
Personally, I think the idea of having huge directories like that inside a project dir to be detrimental. Oh well...
Listen should ignore symlinks - so that could be another workaround. (Although symlink handling is another can of worms).
Personally, I think the idea of having huge directories like that inside a project dir to be detrimental.
It IS detrimental... I'd say it's MENTAL... BAD NODEJS, BAD!
But aside from this, I think it would be a good idea to be able to configure this somehow, don't you?
I'm guessing you can do something like:
Spring.watcher.ignore(/^node_modules$/) if Spring.watcher.respond_to?(:ignore)
@e2 You can, although AFAICT Listen still ends up tracking those files via listen/kqueue, it just doesn't report them to Spring.
This can be pretty slow: if I use @vovimayhem's base_directories
hack to only watch subdirectories, then I can cold-boot spring + my app in 4.5 seconds. If I don't use the hack, and Listen ends up tracking my node_modules files (amongst others), it takes 12 seconds.
That said, the tradeoff is that you can't watch anything that isn't in those directories. So in @vovimayhem's example -
Spring::Watcher::Listen.class_eval do
def base_directories
%w(app config lib spec vendor)
.uniq.map { |path| Pathname.new(File.join(root, path)) }
end
end
%w(
.ruby-version
.rbenv-vars
tmp/restart.txt
tmp/caching-dev.txt
).each { |path| Spring.watch(path) }
AFAICT, none of those paths being passed to Spring.watch
are actually having any effect, since tmp
and the rails root directory aren't included in base_directories
.
Alternatively you can create file ~/.spring.rb
and add code here:
require 'spring/watcher/listen'
Spring::Watcher::Listen.class_eval do
def base_directories
%w(app config lib spec vendor)
.uniq.map { |path| Pathname.new(File.join(root, path)) }
end
end
Actually more correct way is to use "native" listen
's ignore:
option see here or some other "native" options, e.g.:
# e.g. in config/spring.rb
module SpringWatcherListenIgnorer
def start
super
listener.ignore(/^client|^node_modules|^storage/)
end
end
Spring::Watcher::Listen.prepend SpringWatcherListenIgnorer
as a bonus - no class_eval
, correct behaviour for other possible non-ignored folders & files, ability to use more precise configs from listen
.
@dzirtusss I'm not so sure about that - from the docs:
Note: Ignoring paths does not improve performance, except when Polling (#274)
With your config, Listen is observing changes (via kqueue or whatever) to every single file in the rails-root directory & subdirectories, then filtering them out on the ruby side with that regexp. That's a lot of unnecessary work, especially with giant directory trees like node_modules.
@jdelStrother I'm not so sure about that....
Really sorry, but because patch works, this ^^^ means only lack of understanding why this is happening. Without patch, spring threads are on ~100% cpu per thread, with this patch on 0% cpu per thread. Same way as with base_directories patch above.
IMO, this patch only proves that issue is after ignore regex part - more deeper in listener or in spring 🤷♂
Some observations - when it goes into cpu throttling, it goes there in a never-ending way (at least I never noticed it stopped). This is very strange because if this was a qty of changed files issue it should stop eventually, but it keeps rolling cpu for hours. Plus node_modules folders don't change 😄
My best guess - this is qty of files (not qty of changes to files) and some non-optimised looping over it somewhere. Because node_modules type of folders are mostly static folders, some place it just can't handle such qty. I mean I don't say it is that simple, but if I'd need to debug this, I would go this road.
Thank you @dzirtusss! My CPU is idling again! In the end my config/spring.rb
file looks like this:
Spring.watch(
".ruby-version",
".rbenv-vars",
"tmp/restart.txt",
"tmp/caching-dev.txt"
)
module SpringWatcherListenIgnorer
def start
super
listener.ignore(/^data/)
end
end
Spring::Watcher::Listen.prepend SpringWatcherListenIgnorer
It was important to add your code after the Spring.watch
call.
Docs for https://github.com/guard/listen#performance do say:
using :ignore and :only options to avoid tracking directories you don't care about (important with Polling and on MacOS)
Ultra low-level solution to block inotify from listening to some folders.
I have a pgdata folder with a docker volume containing the postgresql databnase, which is not accessible to the current user. inotify was failing with permission error.
INOTIFY_BLACKLIST = %w[
.git
node_modules
pgdata-*
tmp
logs
].map { |dir| Pathname(Spring.watcher.root).glob(dir) }.flatten.map(&:to_s)
require 'rb-inotify'
module INotifyNotifierIgnorer
def watch(path, *flags, &callback)
return if INOTIFY_BLACKLIST.include?(path)
# warn path # to see all the paths listened to
super(path, *flags, &callback)
end
end
INotify::Notifier.prepend INotifyNotifierIgnorer
(in config/spring.rb
before the Spring.watch(...)
)
This will block any other attempt to listen to those directories even from outside of Spring, but unless you are doing something very special, this should not be a problem.
@mildred Thanks! This was the only solution that worked for me.
I went from having more than 200K observers to just about 200.
While working with
rails
version 5, andember-cli-rails
, I end up waiting minutes for the rails console to come up, becausespring-watcher-listen
is observing changes on folders such asbower_components
andnode_modules
, which tend to be huge trees.I had to come by with a nasty workaround - see related issue on thoughtbot/ember-cli-rails - on the
config/spring.rb
file:Is there something to be done to configure this?