Some way to ignore files on Rails 5...

[vovimayhem]

While working with rails version 5, and ember-cli-rails, I end up waiting minutes for the rails console to come up, because spring-watcher-listen is observing changes on folders such as bower_components and node_modules, which tend to be huge trees.

I had to come by with a nasty workaround - see related issue on thoughtbot/ember-cli-rails - on the config/spring.rb file:

Spring::Watcher::Listen.class_eval do
  def base_directories
    %w(app config lib spec vendor)
      .uniq.map { |path| Pathname.new(File.join(root, path)) }
  end
end

%w(
  .ruby-version
  .rbenv-vars
  tmp/restart.txt
  tmp/caching-dev.txt
).each { |path| Spring.watch(path) }

Is there something to be done to configure this?

[e2]

Things like 'bower_components' and 'node_modules' should be ignored by Listen default. (I would've done that a long time ago, except I was worried about SemVer - I was probably too careful).

Listen has a default list of files/directories to ignore. I think both should be added there. I doubt anyone would get hurt if I just do a minor version bump.

Personally, I think the idea of having huge directories like that inside a project dir to be detrimental. Oh well...

Listen should ignore symlinks - so that could be another workaround. (Although symlink handling is another can of worms).

[vovimayhem]

Personally, I think the idea of having huge directories like that inside a project dir to be detrimental.

It IS detrimental... I'd say it's MENTAL... BAD NODEJS, BAD!

But aside from this, I think it would be a good idea to be able to configure this somehow, don't you?

[e2]

I'm guessing you can do something like:

Spring.watcher.ignore(/^node_modules$/) if Spring.watcher.respond_to?(:ignore)

[jdelStrother]

@e2 You can, although AFAICT Listen still ends up tracking those files via listen/kqueue, it just doesn't report them to Spring. This can be pretty slow: if I use @vovimayhem's base_directories hack to only watch subdirectories, then I can cold-boot spring + my app in 4.5 seconds. If I don't use the hack, and Listen ends up tracking my node_modules files (amongst others), it takes 12 seconds.

---

That said, the tradeoff is that you can't watch anything that isn't in those directories. So in @vovimayhem's example -

Spring::Watcher::Listen.class_eval do
  def base_directories
    %w(app config lib spec vendor)
      .uniq.map { |path| Pathname.new(File.join(root, path)) }
  end
end

%w(
  .ruby-version
  .rbenv-vars
  tmp/restart.txt
  tmp/caching-dev.txt
).each { |path| Spring.watch(path) }

AFAICT, none of those paths being passed to Spring.watch are actually having any effect, since tmp and the rails root directory aren't included in base_directories.

[1v]

Alternatively you can create file ~/.spring.rb and add code here:

require 'spring/watcher/listen'

Spring::Watcher::Listen.class_eval do
  def base_directories
    %w(app config lib spec vendor)
      .uniq.map { |path| Pathname.new(File.join(root, path)) }
  end
end

[dzirtusss]

Actually more correct way is to use "native" listen's ignore: option see here or some other "native" options, e.g.:

# e.g. in config/spring.rb
module SpringWatcherListenIgnorer
  def start
    super
    listener.ignore(/^client|^node_modules|^storage/)
  end
end
Spring::Watcher::Listen.prepend SpringWatcherListenIgnorer

as a bonus - no class_eval, correct behaviour for other possible non-ignored folders & files, ability to use more precise configs from listen.

[jdelStrother]

@dzirtusss I'm not so sure about that - from the docs:

Note: Ignoring paths does not improve performance, except when Polling (#274)

With your config, Listen is observing changes (via kqueue or whatever) to every single file in the rails-root directory & subdirectories, then filtering them out on the ruby side with that regexp. That's a lot of unnecessary work, especially with giant directory trees like node_modules.

[dzirtusss]

@jdelStrother I'm not so sure about that....

Really sorry, but because patch works, this ^^^ means only lack of understanding why this is happening. Without patch, spring threads are on ~100% cpu per thread, with this patch on 0% cpu per thread. Same way as with base_directories patch above.

IMO, this patch only proves that issue is after ignore regex part - more deeper in listener or in spring 🤷‍♂

Some observations - when it goes into cpu throttling, it goes there in a never-ending way (at least I never noticed it stopped). This is very strange because if this was a qty of changed files issue it should stop eventually, but it keeps rolling cpu for hours. Plus node_modules folders don't change 😄

My best guess - this is qty of files (not qty of changes to files) and some non-optimised looping over it somewhere. Because node_modules type of folders are mostly static folders, some place it just can't handle such qty. I mean I don't say it is that simple, but if I'd need to debug this, I would go this road.

[eikes]

Thank you @dzirtusss! My CPU is idling again! In the end my config/spring.rb file looks like this:

Spring.watch(
  ".ruby-version",
  ".rbenv-vars",
  "tmp/restart.txt",
  "tmp/caching-dev.txt"
)

module SpringWatcherListenIgnorer
  def start
    super
    listener.ignore(/^data/)
  end
end

Spring::Watcher::Listen.prepend SpringWatcherListenIgnorer

It was important to add your code after the Spring.watch call.

[justin808]

Docs for https://github.com/guard/listen#performance do say:

using :ignore and :only options to avoid tracking directories you don't care about (important with Polling and on MacOS)

[mildred]

Ultra low-level solution to block inotify from listening to some folders.

I have a pgdata folder with a docker volume containing the postgresql databnase, which is not accessible to the current user. inotify was failing with permission error.

INOTIFY_BLACKLIST = %w[
  .git
  node_modules
  pgdata-*
  tmp
  logs
].map { |dir| Pathname(Spring.watcher.root).glob(dir) }.flatten.map(&:to_s)

require 'rb-inotify'
module INotifyNotifierIgnorer
  def watch(path, *flags, &callback)
    return if INOTIFY_BLACKLIST.include?(path)
    # warn path # to see all the paths listened to
    super(path, *flags, &callback)
  end
end
INotify::Notifier.prepend INotifyNotifierIgnorer

(in config/spring.rb before the Spring.watch(...))

This will block any other attempt to listen to those directories even from outside of Spring, but unless you are doing something very special, this should not be a problem.

[juanpabloxk]

@mildred Thanks! This was the only solution that worked for me.

I went from having more than 200K observers to just about 200.