Uberjar Woes!

[cnuernber]

Uberjar's using require-python are currently failing.

We at least need to dig in, investigate, and document how to work around this if not fix it.

[gigasquid]

Here is a minimal project to reproduce https://github.com/gigasquid/libpython-clj-uberjar-test

and the error

08:27 $ java -jar target/libpython-clj-uberjar-test-0.1.0-SNAPSHOT-standalone.jar
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Detecting startup-info for Python executable: 
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Startup info detected: {:python-home "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7", :lib-version "3.7", :libname "python3.7m", :java-library-path-addendum "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib"}
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Setting java library path: /usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib:/Users/cmeier/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Reference thread starting
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Library python3.7m found at [:java-library-path "/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"]
Feb 25, 2020 8:27:35 AM clojure.tools.logging$eval136$fn__139 invoke
INFO: Library c found at [:system "c"]
Exception in thread "main" java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at clojure.lang.RT.classForName(RT.java:2207)
    at clojure.lang.RT.classForName(RT.java:2216)
    at clojure.lang.RT.loadClassForName(RT.java:2235)
    at clojure.lang.RT.load(RT.java:453)
    at clojure.lang.RT.load(RT.java:428)
    at clojure.core$load$fn__6824.invoke(core.clj:6126)
    at clojure.core$load.invokeStatic(core.clj:6125)
    at clojure.core$load.doInvoke(core.clj:6109)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5908)
    at clojure.core$load_one.invoke(core.clj:5903)
    at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
    at clojure.core$load_lib.invokeStatic(core.clj:5947)
    at clojure.core$load_lib.doInvoke(core.clj:5928)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:667)
    at clojure.core$load_libs.invokeStatic(core.clj:5985)
    at clojure.core$load_libs.doInvoke(core.clj:5969)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:667)
    at clojure.core$require.invokeStatic(core.clj:6007)
    at clojure.core$require.doInvoke(core.clj:6007)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at libpython_clj_uberjar_test.core$loading__6706__auto____171.invoke(core.clj:1)
    at libpython_clj_uberjar_test.core__init.load(Unknown Source)
    at libpython_clj_uberjar_test.core__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at clojure.lang.RT.classForName(RT.java:2207)
    at clojure.lang.RT.classForName(RT.java:2216)
    at clojure.lang.RT.loadClassForName(RT.java:2235)
    at clojure.lang.RT.load(RT.java:453)
    at clojure.lang.RT.load(RT.java:428)
    at clojure.core$load$fn__6824.invoke(core.clj:6126)
    at clojure.core$load.invokeStatic(core.clj:6125)
    at clojure.core$load.doInvoke(core.clj:6109)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.lang.Var.invoke(Var.java:384)
    at clojure.lang.Util.loadWithClass(Util.java:250)
    at libpython_clj_uberjar_test.core.<clinit>(Unknown Source)
Caused by: java.lang.ClassCastException: clojure.lang.Var$Unbound cannot be cast to clojure.lang.MultiFn
    at libpython_clj.require__init.load(Unknown Source)
    at libpython_clj.require__init.<clinit>(Unknown Source)
    ... 42 more

[joinr]

Relevant pr here

[metasoarous]

Just wanted to chime in that I had to jump through the hoops in the libpython-clj-uberjar-test repo to get libpython-clj to AOT compile for use in Clojupyter, but was ultimately able to get it working. Hope to be able to make this public soon for people's reference.

Interested to hear whether folks think there's a way to fix this directly in libpython-clj.

Thanks!

[zendevil]

Is there a solution to get the require-python working yet?

[joinr]

@zendevil Did you try using a shim class like in my PR files here? That should get an AOT'd uberjar working by essentially avoiding AOT compiling all the python interop stuff, and late-binding everything as you would in the REPL. So there isn't any deep AOT compilation of the classes, rather a tiny one that loads clojure enough to resolve the namespace (clj file) that it can invoke, and provides an entry point to do this (basically bypass most of the AOT machinery). I do not know what @metasoarous ended up doing with his situation with Clojupyter though.

[metasoarous]

I more or less cribbed your solution exactly @joinr. Again, thanks for forging that trail!

@zendevil If you want to look at another example in context, please see https://github.com/pol-is/analysis

[cnuernber]

@gigasquid @metasoarous @joinr @zendevil - There is now static code generation so you can generate AOT-safe clojure namespaces once and then require them as needed. They do not need python even at require time but only at runtime when the actual var is called the first time.

[metasoarous]

Amazing! Thanks so much for the update on this @cnuernber!