boot-clj / boot

Build tooling for Clojure.
https://boot-clj.github.io/
Eclipse Public License 1.0
1.75k stars 181 forks source link

`boot.pod/make-pod` (and thus `boot.core/aot`) fail with envs that are too large #733

Open alexandergunnarson opened 5 years ago

alexandergunnarson commented 5 years ago

Describe the bug boot.pod/make-pod (and thus its dependents like boot.core/aot) fails with environments that are sufficiently large. I came across this bug while working on a project with a lot of dependencies (760 jars) and thus a :boot-class-path of 83324 chars. The reason for the failure is that environments beyond a certain size cannot be embedded in code within boot.pod/init-pod!: https://github.com/boot-clj/boot/blob/8b5fd047bd1c6fd33438e31062780084aca37dee/boot/pod/src/boot/pod.clj#L850

To Reproduce Steps to reproduce the behavior:

  1. From the REPL (though this bug also occurs from the commandline, it's easier to reproduce at the REPL), run (boot.pod/make-pod env) where env is a "sufficiently large" Boot environment. In my case it failed with a :boot-class-path of 83324 chars (also failed with :fake-class-path), but didn't care about the :dependencies size (it failed when I kept :boot-class-path in the env, whether or not I set :dependencies to []).
  2. Observe that the following exception occurs:
    
    ;; Given a dependency of Clojure 1.10
    java.lang.IllegalArgumentException:
    clojure.lang.Compiler$CompilerException: Syntax error compiling fn* at (0:0).

;; Given a dependency of Clojure 1.8 or 1.9 java.lang.ClassFormatError: Unknown constant tag 112 in class file clojure/core$eval21 clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Unknown constant tag 112 in class file clojure/core$eval21, compiling:(NO_SOURCE_PATH:0:0)


**Expected behavior**
`boot.pod/make-pod` and dependents (`boot.core/aot` among them) should gracefully handle very large Boot environments.

**Desktop**
 - OS: Mac OS 10.14.1
- Java: 1.8

**Additional context**
This is not due to the one classpath string being too big, as I tried splitting it up at compile-time via `partition-all`, embedding that in code, and recombining at runtime.

The workaround I'm using right now is to spit the environment to a file and read it in again:
```clojure
(defn- init-pod!
  [env pod]
  ;; This was added
  (let [temp-file (doto (java.io.File/createTempFile "env" ".edn") (.deleteOnExit))]
    ;; This was added
    (spit temp-file env)
    (doto pod
      (require-in "boot.pod")
      (with-invoke-in (boot.pod/set-worker-pod! worker-pod))
      (with-eval-in
        (require 'boot.util 'boot.pod)
        (reset! boot.util/*verbosity* ~(deref util/*verbosity*))
        ;; This was changed
        (alter-var-root #'boot.pod/env (constantly (slurp ~(.getPath temp-file))))
        (create-ns 'pod)
        (dosync (alter @#'clojure.core/*loaded-libs* conj 'pod))
        (alter-var-root #'*ns* (constantly (the-ns 'pod)))
        (clojure.core/refer-clojure)))))