cnuernber
commented
2 years ago The problem is that both supported dtype-next FFI bindings - JNA and JDK-17 failed. libpython-clj works with the JNA bindings so what happened is the system failed to generate them. Is this jar run in a security context that disables aspects of the JVM that JNA needs to load?
What is the output if your main function calls (tech.v3.datatype.ffi/set-ffi-impl! :jna) before libpython-clj's initialize! function or any require-python calls?
bajrachar
aaelony
I am trying to run a python model inside a hadoop MapReduce task -- so basically a libpython-clj interface within a clj uberjar. When testing within a REPL --> libpython seems to find the python libraries just fine however when run via the hadoop cli it cannot -->
hadoop jar target/hadoop-clj-0.1.0-SNAPSHOT-standalone.jar tmp test.xml2022-02-27 00:43:32,724 INFO HSQLDB742713B661.ENGINE: dataFileCache open start 2022-02-27 00:43:35,518 INFO python.info: Detecting startup info 2022-02-27 00:43:35,799 INFO libpython-clj2.python: Startup info {:lib-version "3.7", :java-library-path-addendum "/home/ravi/.pyenv/versions/3.7.5/lib", :exec-prefix "/home/ravi/.pyenv/versions/3.7.5", :executable "/home/ravi/.pyenv/versions/3.7.5/bin/python3", :libnames ("python3.7m" "python3.7"), :prefix "/home/ravi/.pyenv/versions/3.7.5", :base-prefix "/home/ravi/.pyenv/versions/3.7.5", :libname "python3.7m", :base-exec-prefix "/home/ravi/.pyenv/versions/3.7.5", :python-home "/home/ravi/.pyenv/versions/3.7.5", :version [3 7 5], :platform "linux"} 2022-02-27 00:43:35,799 INFO libpython-clj2.python: Prefixing java library path: /home/ravi/.pyenv/versions/3.7.5/lib 2022-02-27 00:43:35,885 ERROR datatype.ffi: Failed to find a suitable FFI implementation. Attempted both :jdk and :jna -- call set-ffi-impl! from the repl to see specific failure. Syntax error compiling at (tech/v3/datatype/ffi/mmodel.clj:1:1). at clojure.lang.Compiler.load(Compiler.java:7648) at clojure.lang.RT.loadResourceScript(RT.java:381) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.load(RT.java:459) at clojure.lang.RT.load(RT.java:424) at clojure.core$load$fn6839.invoke(core.clj:6126) at clojure.core$load.invokeStatic(core.clj:6125) at clojure.core$load.doInvoke(core.clj:6109) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5908) at clojure.core$load_one.invoke(core.clj:5903) at clojure.core$load_lib$fn6780.invoke(core.clj:5948) at clojure.core$load_lib.invokeStatic(core.clj:5947) at clojure.core$load_lib.doInvoke(core.clj:5928) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:667) at clojure.core$load_libs.invokeStatic(core.clj:5985) at clojure.core$load_libs.doInvoke(core.clj:5969) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:667) at clojure.core$require.invokeStatic(core.clj:6007) at clojure.core$require.doInvoke(core.clj:6007) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:665) at clojure.core$serialized_require.invokeStatic(core.clj:6079) at clojure.core$requiring_resolve.invokeStatic(core.clj:6088) at clojure.core$requiring_resolve.invoke(core.clj:6082) at tech.v3.datatype.ffi$set_ffi_implBANG.invokeStatic(ffi.clj:287) at tech.v3.datatype.ffi$set_ffi_implBANG.invoke(ffi.clj:281) at tech.v3.datatype.ffi$ffi_impl$fn5589$fn__5590.invoke(ffi.clj:302) at tech.v3.datatype.ffi$ffi_impl$fn5589.invoke(ffi.clj:301) at tech.v3.datatype.ffi$ffi_impl.invokeStatic(ffi.clj:296) at tech.v3.datatype.ffi$ffi_impl.invoke(ffi.clj:291) at tech.v3.datatype.ffi$find_library$fn__5600.invoke(ffi.clj:322)