minikomi commented 6 years ago

Boot Bug Report

Platform details

Platform (macOS, Linux, Windows): Mac OSX
Platform version: Sierra 10.12.6
JRE/JDK version: "1.8.0_121"

Boot details

BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.8.0
BOOT_VERSION=2.7.1

`build.boot` content (if applicable)

(set-env!
 :resource-paths #{"resources"})

Description

When there are a reasonably large number of directories in resources, running boot, even without any tasks, causes many errors to the tune of:

2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

More directories (eg. 3000) causes the task to never finish / error out.

                                                            Boot.main                         Boot.java:  257
                                                                  ...
                                                        boot.App.main                          App.java:  491
                                                     boot.App.runBoot                          App.java:  399
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  150
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  159
                                                                  ...
                                                      boot.main/-main                          main.clj:  206
                                                   boot.main/-main/fn                          main.clj:  206
                                                                  ...
                                                     boot.user/eval32  boot.user3887463377436593005.clj:    5
                                                                  ...
                                                   boot.core/set-env!                          core.clj:  873
                                                                  ...
                                                   clojure.core/swap!                          core.clj: 2253
                                                   clojure.core/swap!                          core.clj: 2263
                                                                  ...
                                                   boot.core/init!/fn                          core.clj:  776
                                                boot.core/configure!*                          core.clj:  311
                                                                  ...
                                                         boot.core/fn                          core.clj:  797
                                             boot.core/set-user-dirs!                          core.clj:  196
                                                                  ...
                                                 boot.core/watch-dirs                          core.clj:  744
                                                                  ...
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  117
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  124
                                                                  ...
                                            boot.watcher/make-watcher                       watcher.clj:  125
                                                 boot.watcher/service                       watcher.clj:   94
                                           boot.watcher/service/doreg                       watcher.clj:   93
                                      boot.watcher/register-recursive                       watcher.clj:   65
                                      boot.watcher/register-recursive                       watcher.clj:   62
                                                    boot.watcher/fn/G                       watcher.clj:   25
                                                      boot.watcher/fn                       watcher.clj:   52
              com.barbarysoftware.watchservice.WatchableFile.register                WatchableFile.java:   43
              com.barbarysoftware.watchservice.WatchableFile.register                WatchableFile.java:   34
com.barbarysoftware.watchservice.MacOSXListeningWatchService.register  MacOSXListeningWatchService.java:   52
                                               java.lang.Thread.start                       Thread.java:  714
                                              java.lang.Thread.start0                        Thread.java
java.lang.OutOfMemoryError: unable to create new native thread
clojure.lang.ExceptionInfo: unable to create new native thread
    file: "/var/folders/dm/63qgv3gx0_96qkqbz6xpdzgh0000gn/T/boot.user3887463377436593005.clj"

Steps to reproduce

cd resources
seq 600 | xargs mkdir
cd ..
boot

yannvanhalewyn commented 6 years ago

Thanks for the ref, @arichiardi, removing ~/.boot/cache solved it for me (for now)

arichiardi commented 6 years ago

This also looks like the cache does not evenly distribute files in some corner case. Compared to a poor key choice while sharding maybe.

yannvanhalewyn commented 6 years ago

Care to expand on that? (new to boot) Sadly, it came back after 2-3 restarts and unable to build bigger projects (multi figwheel builds for example). A more permanent solution would be appreciated. Is there a way for me to not have boot monitor the resource paths for example?

dustingetz commented 6 years ago

boot watch --help documents the regex pattern for the watch

dustingetz commented 6 years ago

i solved this by only having package.json in resource-paths and then running "cd target; yarn" after the resources are copied (assuming node_modules was the problematic resource)

yannvanhalewyn commented 6 years ago

Thanks, I was already checking out exclusion options for the watch task and didn't get it to work. Do you know if the repl task uses the watch task behind the scenes? Because I don't when launching the repl

yannvanhalewyn commented 6 years ago

So indeed no watch task was running, that wasn't the issue. I had a stroll through the boot source, and this is what I seem to understand:

Boot syncs src and resources paths out to the tempdirs (filesets)
This happens once when the task starts and then a watcher takes over (this one is causing the FSEvent errors)

Now boot apparently has support for ignore files, .bootignore. Which is a great solution! Sadly these are only taken into account in the first step - the watchers just recursively watch entire directories.

So for now (and future readers) I fixed this issue for me using boot --disable-watchers ..., which is great for figwheel like things, but could potentially break other building blocks. I will open a PR to attempt to use .bootignore regexes in the watchers.

boot-clj / boot

OSX: Reasonably large number of directories causes "FSEventStreamStart: register_with_server: ERROR" #641

Boot Bug Report

Platform details

Boot details

`build.boot` content (if applicable)

Description

Steps to reproduce

boot-clj / boot

OSX: Reasonably large number of directories causes "FSEventStreamStart: register_with_server: ERROR" #641

Boot Bug Report

Platform details

Boot details

build.boot content (if applicable)

Description

Steps to reproduce

`build.boot` content (if applicable)