search

kit-clj / kit

Lightweight, modular framework for scalable web development in Clojure
https://kit-clj.github.io/
MIT License
463 stars 43 forks source link

build error with libpython-clj #107

Closed vinurs closed 11 months ago

vinurs commented 11 months ago

i'm trying use https://github.com/clj-python/libpython-clj in this project, i'm using openjdk version "17.0.7" 2023-04-18, so i add this jvm-opts

image

when i use bb cider it works with python, but when i want to build it using bb uberjar, it outputs the error

bb uberjar
WARNING: Using incubator modules: jdk.incubator.vector, jdk.incubator.foreign
Cleaning target
Writing Pom...
Compiling Clojure...
9590  INFO  libpython-clj2.python.info - Detecting startup info
9918  INFO  libpython-clj2.python - Startup info {:lib-version "3.10", :java-library-path-addendum "/Users/vinurs/.pyenv/versions/3.10.3/lib", :exec-prefix "/Users/vinurs/.pyenv/versions/3.10.3", :executable "/Users/vinurs/.pyenv/versions/3.10.3/bin/python3", :libnames ("python3.10m" "python3.10"), :prefix "/Users/vinurs/.pyenv/versions/3.10.3", :base-prefix "/Users/vinurs/.pyenv/versions/3.10.3", :libname "python3.10m", :base-exec-prefix "/Users/vinurs/.pyenv/versions/3.10.3", :python-home "/Users/vinurs/.pyenv/versions/3.10.3", :version [3 10 3], :platform "darwin"}
9918  INFO  libpython-clj2.python - Prefixing java library path: /Users/vinurs/.pyenv/versions/3.10.3/lib
9985  INFO  libpython-clj2.python - Loading python library: /Users/vinurs/.pyenv/versions/3.10.3/lib/libpython3.10.dylib
10361 INFO  tech.v3.resource.gc - Reference thread starting
10390 INFO  tech.v3.datatype.nio-buffer - Unable to find direct buffer constructor -
falling back to jdk16 memory model.
10392 ERROR libpython-clj2.python.ffi - Exception while releasing object
java.lang.RuntimeException: Unable to load direct buffer constructor.  If you are using JDK-17, set your runtime :jvm-opts as follows:
:jvm-opts ["--add-modules" "jdk.incubator.foreign,jdk.incubator.vector"
                         "--enable-native-access=ALL-UNNAMED"]}
        at tech.v3.datatype.nio_buffer$fn__37513$fn__37516.invoke(nio_buffer.clj:122)
        at tech.v3.datatype.nio_buffer$fn__37513.invokeStatic(nio_buffer.clj:121)
        at tech.v3.datatype.nio_buffer$fn__37513.invoke(nio_buffer.clj:109)
        at clojure.lang.Delay.deref(Delay.java:42)
        at clojure.core$deref.invokeStatic(core.clj:2337)
        at clojure.core$deref.invoke(core.clj:2323)
        at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invokeStatic(nio_buffer.clj:135)
        at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invoke(nio_buffer.clj:130)
        at libpython_clj2.python.ffi$pystr__GT_str.invokeStatic(ffi.clj:798)
        at libpython_clj2.python.ffi$pystr__GT_str.invoke(ffi.clj:786)
        at libpython_clj2.python.ffi$pytype_name.invokeStatic(ffi.clj:806)
        at libpython_clj2.python.ffi$pytype_name.invoke(ffi.clj:802)
        at libpython_clj2.python.ffi$pyobject_type_kwd$reify__37847.apply(ffi.clj:822)
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
        at libpython_clj2.python.ffi$pyobject_type_kwd.invokeStatic(ffi.clj:818)
        at libpython_clj2.python.ffi$pyobject_type_kwd.invoke(ffi.clj:815)
        at libpython_clj2.python.ffi$simplify_or_track.invokeStatic(ffi.clj:962)
        at libpython_clj2.python.ffi$simplify_or_track.invoke(ffi.clj:954)
        at libpython_clj2.python.fn$call_py_fn.invokeStatic(fn.clj:199)
        at libpython_clj2.python.fn$call_py_fn.invoke(fn.clj:179)
        at libpython_clj2.python.fn$fn__38464.invokeStatic(fn.clj:205)
        at libpython_clj2.python.fn$fn__38464.invoke(fn.clj:202)
        at libpython_clj2.python.protocols$fn__38067$G__38060__38076.invoke(protocols.clj:67)
        at libpython_clj2.python.fn$call.invokeStatic(fn.clj:213)
        at libpython_clj2.python.fn$call.doInvoke(fn.clj:210)
        at clojure.lang.RestFn.invoke(RestFn.java:460)
        at libpython_clj2.python.class$create_class$fn__38672.invoke(class.clj:83)
        at libpython_clj2.python.class$create_class.invokeStatic(class.clj:80)
        at libpython_clj2.python.class$create_class.invoke(class.clj:72)
        at libpython_clj2.python.io_redirect$fn__39345$fn__39346.invoke(io_redirect.clj:23)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at libpython_clj2.python.io_redirect$fn__39345.invokeStatic(io_redirect.clj:22)
        at libpython_clj2.python.io_redirect$fn__39345.invoke(io_redirect.clj:22)
        at clojure.lang.Delay.deref(Delay.java:42)
        at clojure.core$deref.invokeStatic(core.clj:2337)
        at clojure.core$deref.invoke(core.clj:2323)
        at libpython_clj2.python.io_redirect$setup_std_writer$fn__39355.invoke(io_redirect.clj:50)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at libpython_clj2.python.io_redirect$setup_std_writer.invokeStatic(io_redirect.clj:48)
        at libpython_clj2.python.io_redirect$setup_std_writer.invoke(io_redirect.clj:44)
        at libpython_clj2.python.io_redirect$redirect_io_BANG_.invokeStatic(io_redirect.clj:58)
        at libpython_clj2.python.io_redirect$redirect_io_BANG_.invoke(io_redirect.clj:56)
        at libpython_clj2.python$initialize_BANG_$fn__39377.invoke(python.clj:149)
        at libpython_clj2.python$initialize_BANG_.invokeStatic(python.clj:142)
        at libpython_clj2.python$initialize_BANG_.doInvoke(python.clj:45)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3714)
        at clojure.lang.Compiler.compile1(Compiler.java:7748)
        at clojure.lang.Compiler.compile(Compiler.java:7815)
        at clojure.lang.RT.compile(RT.java:411)
        at clojure.lang.RT.load(RT.java:457)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6908.invoke(core.clj:6161)
        at clojure.core$load.invokeStatic(core.clj:6160)
        at clojure.core$load.doInvoke(core.clj:6144)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5933)
        at clojure.core$load_one.invoke(core.clj:5928)
        at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)
        at clojure.core$load_lib.invokeStatic(core.clj:5974)
        at clojure.core$load_lib.doInvoke(core.clj:5953)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:6016)
        at clojure.core$load_libs.doInvoke(core.clj:6000)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:6038)
        at clojure.core$require.doInvoke(core.clj:6038)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at libpython_clj2.require$loading__6789__auto____29935.invoke(require.clj:1)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3714)
        at clojure.lang.Compiler.compile1(Compiler.java:7748)
        at clojure.lang.Compiler.compile1(Compiler.java:7738)
        at clojure.lang.Compiler.compile(Compiler.java:7815)
        at clojure.lang.RT.compile(RT.java:411)
        at clojure.lang.RT.load(RT.java:457)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6908.invoke(core.clj:6161)
        at clojure.core$load.invokeStatic(core.clj:6160)
        at clojure.core$load.doInvoke(core.clj:6144)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5933)
        at clojure.core$load_one.invoke(core.clj:5928)
        at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)
        at clojure.core$load_lib.invokeStatic(core.clj:5974)
        at clojure.core$load_lib.doInvoke(core.clj:5953)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:6016)
        at clojure.core$load_libs.doInvoke(core.clj:6000)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:6038)
        at clojure.core$require.doInvoke(core.clj:6038)
        at clojure.lang.RestFn.invoke(RestFn.java:3204)
        at server.app.core$loading__6789__auto____138.invoke(core.clj:1)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3714)
        at clojure.lang.Compiler.compile1(Compiler.java:7748)
        at clojure.lang.Compiler.compile1(Compiler.java:7738)
        at clojure.lang.Compiler.compile(Compiler.java:7815)
        at clojure.lang.RT.compile(RT.java:411)
        at clojure.lang.RT.load(RT.java:457)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6908.invoke(core.clj:6161)
        at clojure.core$load.invokeStatic(core.clj:6160)
        at clojure.core$load.doInvoke(core.clj:6144)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5933)
        at clojure.core$compile$fn__6913.invoke(core.clj:6171)
        at clojure.core$compile.invokeStatic(core.clj:6171)
        at clojure.core$compile.invoke(core.clj:6163)
        at user$eval136.invokeStatic(compile.clj:6)
        at user$eval136.invoke(compile.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7194)
        at clojure.lang.Compiler.load(Compiler.java:7653)
        at clojure.lang.Compiler.loadFile(Compiler.java:7591)
        at clojure.main$load_script.invokeStatic(main.clj:475)
        at clojure.main$script_opt.invokeStatic(main.clj:535)
        at clojure.main$script_opt.invoke(main.clj:530)
        at clojure.main$main.invokeStatic(main.clj:664)
        at clojure.main$main.doInvoke(main.clj:616)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.main.main(main.java:40)
Caused by: clojure.lang.Compiler$CompilerException: Syntax error macroexpanding at (nio_buf_mmodel.clj:1:1).
        at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3719)
        at clojure.lang.Compiler.compile1(Compiler.java:7748)
        at clojure.lang.Compiler.compile1(Compiler.java:7738)
        at clojure.lang.Compiler.compile(Compiler.java:7815)
        at clojure.lang.RT.compile(RT.java:411)
        at clojure.lang.RT.load(RT.java:457)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6908.invoke(core.clj:6161)
        at clojure.core$load.invokeStatic(core.clj:6160)
        at clojure.core$load.doInvoke(core.clj:6144)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5933)
        at clojure.core$load_one.invoke(core.clj:5928)
        at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)
        at clojure.core$load_lib.invokeStatic(core.clj:5974)
        at clojure.core$load_lib.doInvoke(core.clj:5953)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:6016)
        at clojure.core$load_libs.doInvoke(core.clj:6000)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:6038)
        at clojure.core$require.doInvoke(core.clj:6038)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$serialized_require.invokeStatic(core.clj:6114)
        at clojure.core$requiring_resolve.invokeStatic(core.clj:6123)
        at clojure.core$requiring_resolve.invoke(core.clj:6117)
        ... 141 common frames omitted
Caused by: java.lang.ClassNotFoundException: jdk.incubator.foreign.MemoryAddress
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
        at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at clojure.lang.RT.classForName(RT.java:2209)
        at clojure.lang.RT.classForNameNonLoading(RT.java:2222)
        at tech.v3.datatype.ffi.nio_buf_mmodel$loading__6789__auto____41816.invoke(nio_buf_mmodel.clj:1)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3714)
        ... 169 common frames omitted
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:445).
jdk.incubator.foreign.MemoryAddress

Full report at:
/var/folders/5r/g9bcl2wd68j8v611zphm76jr0000gn/T/clojure-14570190945988579571.edn
Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:103).
Clojure compilation failed, working dir preserved: /var/folders/5r/g9bcl2wd68j8v611zphm76jr0000gn/T/compile-clj6952101937453854079

Full report at:
/var/folders/5r/g9bcl2wd68j8v611zphm76jr0000gn/T/clojure-6935853039996426287.edn
Error while executing task: uberjar

but i've add the

 :jvm-opts ["--add-modules" "jdk.incubator.foreign,jdk.incubator.vector"
                         "--enable-native-access=ALL-UNNAMED"]}
yogthos commented 11 months ago

The jvm-opts are a runtime flag, so you'd have to supply them when you're running the jar as well.

vinurs commented 11 months ago

@yogthos how can i fix this for bb uberjar

yogthos commented 11 months ago

The module gets loaded from the JVM runtime, so it's bnot part of the jar. You just have to pass the flags to the JVM when running the uberjar.

vinurs commented 11 months ago

Now the problem is, when i add require in some clj file like this

(:require [libpython-clj2.require :refer [require-python]]
            [libpython-clj2.python :refer [py. py.. py.-] :as py]
            )

then i can't use bb uberjar to generate the jar， Just prompt me the error above

vinurs commented 11 months ago

i modified build.clj, and add :java-opts fix it

--- a/build.clj
+++ b/build.clj
@@ -38,6 +38,7 @@
   (println "Compiling Clojure...")
   (b/compile-clj {:base base
                   :src-dirs: ["src/clj", "resources", "env/prod/resources", "env/prod/clj"]
+ :java-opts["--add-modules", "jdk.incubator.foreign,jdk.incubator.vector": "--enable-native-access=ALL-UNNAMED"]
                   :class-dir class-dir})
   (println "Making uberjar...")
yogthos commented 11 months ago

Ah yeah that works.