Call to unbound hooks fn

When running

lein monolith each :parallel 6 install

it was progressing through the topo graph building and when

Applying to xxx/yyy-yyyy-yyyy

it encountered an error

java.lang.IllegalStateException: Attempting to call unbound fn: #'lein-environ.plugin/hooks
 at clojure.lang.Var$Unbound.throwArity (Var.java:43)
    clojure.lang.AFn.invoke (AFn.java:28)
    clojure.lang.Var.invoke (Var.java:375)
    leiningen.core.project$load_hook.invokeStatic (project.clj:726)
    leiningen.core.project$load_hook.invoke (project.clj:721)
    leiningen.core.project$load_hooks.invokeStatic (project.clj:740)
    leiningen.core.project$load_hooks.doInvoke (project.clj:733)
    clojure.lang.RestFn.invoke (RestFn.java:410)
    leiningen.core.project$activate_middleware.invokeStatic (project.clj:777)
    leiningen.core.project$activate_middleware.invoke (project.clj:773)
    leiningen.core.project$set_profiles.invokeStatic (project.clj:862)
    leiningen.core.project$set_profiles.doInvoke (project.clj:855)
    clojure.lang.RestFn.invoke (RestFn.java:442)
    leiningen.core.project$merge_profiles.invokeStatic (project.clj:870)
    leiningen.core.project$merge_profiles.invoke (project.clj:864)
    leiningen.jar$warn_implicit_aot.invokeStatic (jar.clj:238)
    leiningen.jar$warn_implicit_aot.invoke (jar.clj:237)
    leiningen.jar$add_main.invokeStatic (jar.clj:249)
    leiningen.jar$add_main.invoke (jar.clj:248)
    leiningen.jar$process_project.invokeStatic (jar.clj:267)
    leiningen.jar$process_project.doInvoke (jar.clj:259)
    clojure.lang.RestFn.invoke (RestFn.java:467)
    leiningen.jar$preprocess_project.invokeStatic (jar.clj:270)
    leiningen.jar$preprocess_project.doInvoke (jar.clj:269)
    clojure.lang.RestFn.invoke (RestFn.java:423)
    leiningen.jar$jar.invokeStatic (jar.clj:342)
    leiningen.jar$jar.invoke (jar.clj:325)
    leiningen.jar$jar.invokeStatic (jar.clj:345)
    leiningen.jar$jar.invoke (jar.clj:325)
    leiningen.install$install.invokeStatic (install.clj:19)
    leiningen.install$install.invoke (install.clj:12)
    clojure.lang.Var.invoke (Var.java:379)
    clojure.lang.AFn.applyToHelper (AFn.java:154)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invokeStatic (core.clj:648)
    clojure.core$apply.invoke (core.clj:641)
    leiningen.core.main$partial_task$fn__5932.doInvoke (main.clj:272)
    clojure.lang.RestFn.invoke (RestFn.java:410)
    clojure.lang.AFn.applyToHelper (AFn.java:154)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:648)
    clojure.core$apply.invoke (core.clj:641)
    leiningen.core.main$apply_task.invokeStatic (main.clj:322)
    leiningen.core.main$apply_task.invoke (main.clj:308)
    leiningen.core.main$resolve_and_apply.invokeStatic (main.clj:328)
    leiningen.core.main$resolve_and_apply.invoke (main.clj:324)
    lein_monolith.task.each$apply_subproject_task.invokeStatic (each.clj:126)
    lein_monolith.task.each$apply_subproject_task.invoke (each.clj:110)
    lein_monolith.task.each$run_task_BANG_.invokeStatic (each.clj:141)
    lein_monolith.task.each$run_task_BANG_.invoke (each.clj:129)
    lein_monolith.task.each$run_parallel_BANG_$future_builder__3484$task_runner__3486$f__2614__auto____3487.invoke (each.clj:190)
    clojure.lang.AFn.run (AFn.java:22)
    io.aleph.dirigiste.Executor$3.run (Executor.java:308)
    io.aleph.dirigiste.Executor$Worker$1.run (Executor.java:62)
    manifold.executor$thread_factory$reify__2496$f__2497.invoke (executor.clj:44)
    clojure.lang.AFn.run (AFn.java:22)
    java.lang.Thread.run (Thread.java:745)

I tried again without :parallel and it succeeded. Trying again with :parallel caused it to recur, so perhaps it is relatively reproducible.

I'm using 0.3.0:

$ cat ~/.lein/profiles.clj
{:user {:plugins [[lein-monolith "0.3.0"]]}}

This still crops up occasionally, so I did some more digging. I believe what is happening is the following:

When lein-monolith is running a higher-order each task with the :parallel option, it constructs a thread pool and calls apply-subproject-task on each project as the upstream projects complete.
Ultimately, this takes two steps for each project - first, init-project is called to prepare the project for the task execution, then resolve-and-apply-task to run the task in the project.
Initializing the project calls activate-middleware which is responsible for (among other things) loading plugin hooks with load-hooks.
To ensure the code for a hook is loaded, leiningen uses an internal require-resolve utility, which predates the requiring-resolve core function added in Clojure 1.10.
Leiningen's require-resolve function first checks whether the namespace of the symbol exists and is loaded before it resolves the symbol, requiring it if not.
Clojure’s require ultimately calls load-lib which uses clojure.lang.RT/load under the hood. That does a bunch of stuff, but importantly the loads aren’t synchronized across threads.
When a namespace is loaded, the compiler emits bytecode that first defines the var and later binds its root to the provided init expression.
In rare cases, it’s possible that a plugin with hooks such as lein-environ is not loaded yet, and two projects using it get run in parallel. Both of them try to load the plugin hooks, one of them goes first and starts compiling the namespace, while the other one sees that the namespace exists and resolves the var before it’s been fully defined.

Clojure 1.10 added a new serialized-require function which is used internally by requiring-resolve, which would address this problem. However, Leiningen is probably not ready to require Clojure 1.10 features, so we'll need to figure out a way to lock appropriately within lein-monolith to fix this.

amperity / lein-monolith

Call to unbound hooks fn #15