search

boot-clj / boot

Build tooling for Clojure.
https://boot-clj.github.io/
Eclipse Public License 1.0
1.75k stars 180 forks source link

java.lang.NoClassDefFoundError: clojure/java/api/Clojure in project that works with Leiningen #665

Closed zilti closed 6 years ago

zilti commented 6 years ago

Description

For all I know, the build.boot below should be functionally equivalent to this project.clj:

(defproject ch.unibe.psy/videocapture "0.9.0"
  :dependencies [[org.clojure/clojure "1.9.0"]
                 [clojurefx "0.5.0-SNAPSHOT" :exclusions [org.clojure/tools.reader]]
                 [org.freedesktop.gstreamer/gst1-java-core "0.9.1"]
                 [net.java.dev.jna/jna "4.5.0"]
                 [org.codelibs/jcifs "1.3.18.3"]
                 [clj-time "0.14.2"]
                 [com.taoensso/timbre "4.10.0" :exclusions [org.clojure/tools.reader]]
                 [org.clojure/core.async "0.3.465" :exclusions [org.ow2.asm/asm-all]]
                 [de.jensd/fontawesomefx "8.9"]
                 [org.controlsfx/controlsfx "8.40.14"]
                 [eu.mihosoft.vrl.workflow/vworkflows-core "0.2.4.3"]
                 [eu.mihosoft.vrl.workflow/vworkflows-fx "0.2.4.3"]]
  :source-paths ["src"]
  :java-source-paths ["src"]
  :resource-paths ["resources"]
  :main videocapture.core)

But it isn't. First of all, I don't need the :exclusions [org.ow2.asm/asm-all] in the Leiningen project; it only leads to an exception using Boot. The exception being that ClojureFX can't find a field that is defined in ASM.

Now, having this out of the way, it's not finished yet. I have an FXML file that defines an org.controlsfx.control.StatusBar. This FXML gets loaded by ClojureFX and then, ClojureFX should return a Scene. But this only works in Leiningen. In Boot, Java throws a fit claiming it can't find the class. It's completely irrelevant if I import the class in build.boot or the main namespace; it just won't work in Boot.

I checked the Boot classpath using boot show -c, and the controlsfx jar is definitely on there.

Environment information

JRE/JDK version (java -version): openjdk version "1.8.0_151"

;; build.boot
(set-env!
 :source-paths #{"src"}
 :resource-paths #{"resources"}
 :dependencies '[[org.clojure/clojure "1.9.0"]
                 [clojurefx "0.5.0-SNAPSHOT" :exclusions [org.clojure/tools.reader]]
                 [org.freedesktop.gstreamer/gst1-java-core "0.9.1"]
                 [net.java.dev.jna/jna "4.5.0"]
                 [org.codelibs/jcifs "1.3.18.3"]
                 [clj-time "0.14.2"]
                 [com.taoensso/timbre "4.10.0" :exclusions [org.clojure/tools.reader]]
                 [org.clojure/core.async "0.3.465" :exclusions [org.ow2.asm/asm-all]]
                 [de.jensd/fontawesomefx "8.9"]
                 [org.controlsfx/controlsfx "8.40.14"]
                 [eu.mihosoft.vrl.workflow/vworkflows-core "0.2.4.3"]
                 [eu.mihosoft.vrl.workflow/vworkflows-fx "0.2.4.3"]])

(deftask run-core []
  (with-pass-thru _
    (require 'videocapture.core)
    ((resolve 'videocapture.core/-main))))

(deftask run
  ""
  []
  (comp (javac)
        (run-core)))
zilti commented 6 years ago

I suspect Boot is doing some weird classpath trickery. The problem seems to be that ClojureFX doesn't depend on controlsfx, but is given an input file by videocapture that needs controlsfx; thus it relies on controlsfx being on the classpath, which should be the case because videocapture depends on it. But boot seems to have other plans.

Here's the Boot dependency tree:

[adzerk/bootlaces "0.1.13" :scope "test"]
[clj-time "0.14.2"]
└── [joda-time "2.9.7"]
[clojurefx "0.5.0-20171212.020734-2" :exclusions [[org.clojure/tools.reader]]]
├── [camel-snake-kebab "0.4.0"]
├── [clojure-jsr-223 "0.1.0"]
├── [net.openhft/compiler "2.3.0"]
│   ├── [com.intellij/annotations "12.0"]
│   └── [org.slf4j/slf4j-api "1.7.14"]
├── [org.ow2.asm/asm-util "6.0"]
│   └── [org.ow2.asm/asm-tree "6.0"]
├── [org.ow2.asm/asm "6.0"]
└── [swiss-arrows "1.0.0"]
[com.taoensso/timbre "4.10.0" :exclusions [[org.clojure/tools.reader]]]
├── [com.taoensso/encore "2.91.0"]
│   └── [com.taoensso/truss "1.5.0"]
└── [io.aviso/pretty "0.1.33"]
[de.jensd/fontawesomefx "8.9"]
[eu.mihosoft.vrl.workflow/vworkflows-core "0.2.4.3"]
├── [com.thoughtworks.xstream/xstream "1.4.8"]
│   ├── [xmlpull "1.1.3.1"]
│   └── [xpp3/xpp3_min "1.1.4c"]
├── [net.sf.jung/jung-algorithms "2.0.1"]
│   └── [colt "1.2.0"]
├── [net.sf.jung/jung-api "2.0.1"]
│   └── [net.sourceforge.collections/collections-generic "4.01"]
├── [net.sf.jung/jung-graph-impl "2.0.1"]
├── [net.sf.jung/jung-visualization "2.0.1"]
└── [org.apache.commons/commons-collections4 "4.1"]
[eu.mihosoft.vrl.workflow/vworkflows-fx "0.2.4.3"]
├── [eu.mihosoft.jfx.scaledfx/scaledfx "0.3"]
├── [eu.mihosoft.jfx.scaledfx/scaledfx "0.3" :classifier "sources"]
└── [org.apache.commons/commons-math3 "3.3"]
[net.java.dev.jna/jna "4.5.0"]
[org.clojure/clojure "1.9.0"]
├── [org.clojure/core.specs.alpha "0.1.24"]
└── [org.clojure/spec.alpha "0.1.143"]
[org.clojure/core.async "0.3.465" :exclusions [[org.ow2.asm/asm-all]]]
└── [org.clojure/tools.analyzer.jvm "0.7.0"]
    ├── [org.clojure/core.memoize "0.5.9"]
    │   └── [org.clojure/core.cache "0.6.5"]
    │       └── [org.clojure/data.priority-map "0.0.7"]
    ├── [org.clojure/tools.analyzer "0.6.9"]
    └── [org.clojure/tools.reader "1.0.0-beta4"]
[org.codelibs/jcifs "1.3.18.3"]
└── [javax.servlet/servlet-api "2.4"]
[org.controlsfx/controlsfx "8.40.14"]
[org.freedesktop.gstreamer/gst1-java-core "0.9.1"]
martinklepsch commented 6 years ago

This is the full stacktrace which can be generated by running boot -vv run in this repo, clone with:

git clone git@bitbucket.org:snippets/zilti/zenrAb/boot-dependencies-bug.git
Exception in Application start method
clojure.lang.ExceptionInfo: Exception in Application start method {:line 36}
    at clojure.core$ex_info.invokeStatic(core.clj:4739)
    at clojure.core$ex_info.invoke(core.clj:4739)
    at boot.main$_main$fn__1201.invoke(main.clj:222)
    at boot.main$_main.invoke(main.clj:216)
    at clojure.lang.Var.invoke(Var.java:396)
    at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:159)
    at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:150)
    at boot.App.runBoot(App.java:399)
    at boot.App.main(App.java:491)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at Boot.main(Boot.java:257)
Caused by: java.lang.RuntimeException: Exception in Application start method
    at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:917)
    at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$155(LauncherImpl.java:182)
    at java.lang.Thread.run(Thread.java:745)
Caused by: javafx.fxml.LoadException:
/Users/martin/etc/boot/cache/tmp/Users/martin/code/04-repro/boot-dependencies-bug/1miy/-ueasqk/

    at javafx.fxml.FXMLLoader.constructLoadException(FXMLLoader.java:2601)
    at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2571)
    at javafx.fxml.FXMLLoader.load(FXMLLoader.java:2425)
    at clojurefx.fxml$load_fxml.invokeStatic(fxml.clj:8)
    at clojurefx.fxml$load_fxml.invoke(fxml.clj:5)
    at clojurefx.fxml$load_fxml_with_controller.invokeStatic(fxml.clj:14)
    at clojurefx.fxml$load_fxml_with_controller.invoke(fxml.clj:12)
    at bootbug.core$start.invokeStatic(core.clj:13)
    at bootbug.core$start.invoke(core.clj:12)
    at clojurefx.ApplicationInitializer.start(ApplicationInitializer.java:19)
    at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$162(LauncherImpl.java:863)
    at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$175(PlatformImpl.java:326)
    at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
    at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
    at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2566)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: clojure/java/api/Clojure
    at ch.lyrion.Test.initialize(Unknown Source)
    ... 25 more
Caused by: java.lang.ClassNotFoundException: clojure.java.api.Clojure
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 26 more
tobias commented 6 years ago

When I run the sample app, I get a different failure:

clojure.lang.ExceptionInfo: clojure/lang/Tuple {:line 24}
    at clojure.core$ex_info.invoke(core.clj:4593)
    at boot.main$_main$fn__1201.invoke(main.clj:222)
    at boot.main$_main.invoke(main.clj:216)
    at clojure.lang.Var.invoke(Var.java:394)
    at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:159)
    at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:150)
    at boot.App.runBoot(App.java:399)
    at boot.App.main(App.java:491)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at Boot.main(Boot.java:258)
Caused by: java.lang.NoClassDefFoundError: clojure/lang/Tuple
    at bootbug.core__init.__init0(Unknown Source)
    at bootbug.core__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at clojure.lang.RT.classForName(RT.java:2154)
    at clojure.lang.RT.classForName(RT.java:2163)
    at clojure.lang.RT.loadClassForName(RT.java:2182)
    at clojure.lang.RT.load(RT.java:436)
    at clojure.lang.RT.load(RT.java:412)
    at clojure.core$load$fn__5448.invoke(core.clj:5866)
    at clojure.core$load.doInvoke(core.clj:5865)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5671)
    at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
    at clojure.core$load_lib.doInvoke(core.clj:5710)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:632)
    at clojure.core$load_libs.doInvoke(core.clj:5749)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:632)
    at clojure.core$require.doInvoke(core.clj:5832)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at boot.user$eval39$fn__40$fn__45$fn__46.invoke(Unknown Source)
    at boot.task.built_in$fn__2547$fn__2548$fn__2555$fn__2556.invoke(built_in.clj:789)
    at boot.task.built_in$fn__2586$fn__2587$fn__2592$fn__2593.invoke(built_in.clj:804)
    at boot.core$run_tasks.invoke(core.clj:1021)
    at boot.core$boot$fn__933.invoke(core.clj:1031)
    at clojure.core$binding_conveyor_fn$fn__4444.invoke(core.clj:1916)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: clojure.lang.Tuple
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 33 more

I also see the following in the -vv output:

Classpath conflict: org.clojure/clojure version 1.7.0 already loaded, NOT loading version 1.9.0

I think the root cause here may be that clojurefx is AOT'ed, and the version of Clojure it was compiled against isn't the version that is getting loaded, possibly.

zilti commented 6 years ago

I just tried running it with a non-AOT'ed ClojureFX, which still results in Caused by: java.lang.NoClassDefFoundError: clojure/java/api/Clojure or, in my other project, java.lang.NoClassDefFoundError: Lorg/controlsfx/control/StatusBar;, respectively.

tobias commented 6 years ago

This is definitely something classloadery, and not AOT related (thanks for confirming). I'll need to take a closer look at what JavaFX is doing/what expectations it has about the effective classpath.

tobias commented 6 years ago

I dug in to this a bit, and I see what's happening. clojurefx is using FXClassLoader/loadClass to define the new class in a classloader. FXClassLoader/loadClass makes the protected defineClass method on the system classloader accessible, and calls that (and that's all it does). The problem there is under boot, the system class loader can't see Clojure, since it is loaded by a lower classloader, so trying to initialize an instance of this new class fails, since it needs Clojure. This doesn't happen under lein (or from an uberjar) because Clojure is visible to the system classloader.

I was able to get this to work under boot with a modification to clojurefx:

diff -r 8ebb6ec132ed src/clojurefx/controllergen.clj
--- a/src/clojurefx/controllergen.clj   Mon Dec 11 20:39:37 2017 +0100
+++ b/src/clojurefx/controllergen.clj   Fri Dec 15 17:38:09 2017 -0500
@@ -173,6 +173,16 @@
     (.visitEnd handled-class)
     (.toByteArray handled-class)))

+(defn define-class [name bytes]
+  (let [cl (.getClassLoader clojure.lang.RT)
+        method (.getDeclaredMethod java.lang.ClassLoader "defineClass"
+                                   (into-array [String (Class/forName "[B") (Integer/TYPE) (Integer/TYPE)]))]
+    (try
+      (.setAccessible method true)
+      (.invoke method cl (into-array Object [name bytes (int 0) (int (count bytes))]))
+      (finally
+        (.setAccessible method false)))))
+
 ;; ;; Plumber

 (defn gen-fx-controller-class [fxmlpath clj-fn]
@@ -186,5 +196,5 @@
     (try
       (Class/forName (str pkg "." classname))
       (catch Exception e
-        (FXClassLoader/loadClass (str pkg "." classname)
-                                 (gen-fx-controller fxmlzip fxmlpath cljvec [pkg classname]))))))
+        (define-class (str pkg "." classname)
+          (gen-fx-controller fxmlzip fxmlpath cljvec [pkg classname]))))))

That just mimics what FXClassLoader/loadClass does, but does it instead to whatever classloader loaded Clojure. I don't know much about JavaFX, but this should be a safe change to make, unless the created classes need to be visible (loadable) by something that is loaded by a higher classloader.

EDIT: I just noticed that FXClassLoader is part of the clojurefx project, and not from JavaFX (I navigated to it in Cursive, so didn't notice the source). You could instead modify its behavior instead of using my Clojure implementation.

zilti commented 6 years ago

Ah, so I used the wrong classloader. Well, I fixed this now. Thank you a lot for helping out!

All that remains now is that I need :exclusions [org.ow2.asm/asm-all] in my build.boot as described in the initial post, but I can live with that, and it's probably not a bug, just a difference in how boot works vs lein.

martinklepsch commented 6 years ago

Thanks for taking a look at this again @tobias :)

@zilti I will close this issue now but if you feel like it should stay open just let leave a note :)