Closed madeinquant closed 2 years ago
(py/initialize!)
should not have returned :already-initialized unless initialize was already called somewhere else. I have seen that happen in embedded mode where it is correct but not in a normal repl mode.
Here is the check for if the system is initialized. How is this check returning true in your case?
The metadata class fails to load often when python itself fails to initialize or when it can't find some required python modules.
Thanks for your comment. fixed with your suggested reference.
Uninstall java 17 with brew
brew uninstall --cask graalvm/tap/graalvm-ce-java17
Install java 11 with brew
brew install --cask graalvm/tap/graalvm-ce-java11
running the following command on os x
xattr -r -d com.apple.quarantine /Library/Java/JavaVirtualMachines/graalvm-ce-java11-22.0.0.2
change your GRAALVM_HOME
and PATH
in ~/.zshrc
export GRAALVM_HOME=/Library/Java/JavaVirtualMachines/graalvm-ce-java11-22.0.0.2/Contents/Home
export PATH=$GRAALVM_HOME/bin:$PATH
Enable your virtual environment and install the packages with pip
pip install numpy torch transformers lime
run lein repl to check the correctness :
libpython-clj-test.core=> (require '[libpython-clj2.python :as py :refer [py. py.. py.-]])
nil
libpython-clj-test.core=> (require '[libpython-clj2.require :refer [require-python]])
nil
libpython-clj-test.core=> (py/initialize!)
:already-initialized
libpython-clj-test.core=> (def np (py/import-module "numpy"))
#'libpython-clj-test.core/np
libpython-clj-test.core=> (py/py. np linspace 2 3 :num 10)
[2. 2.11111111 2.22222222 2.33333333 2.44444444 2.55555556
2.66666667 2.77777778 2.88888889 3. ]
OK, that is a great and nice work.
I think JDK-17 would have worked for you but it requires some extra command line arguments and there should have been quite a bit more logging.
One thing that is confusing to me is I don't see any logging from your output whatsoever. For instance, here is what happens on my computer when I do a similar pathway:
user> (require '[libpython-clj2.python :as py])
nil
user> (require '[libpython-clj2.require :refer [require-python]))
Syntax error reading source at (REPL:45:65).
Unmatched delimiter: )
user> (require '[libpython-clj2.require :refer [require-python]])
Feb 10, 2022 7:09:57 AM clojure.tools.logging$eval5863$fn__5866 invoke
INFO: Detecting startup info
Feb 10, 2022 7:09:57 AM clojure.tools.logging$eval5863$fn__5866 invoke
INFO: Startup info {:lib-version "3.9", :java-library-path-addendum "/home/chrisn/miniconda3/lib", :exec-prefix "/home/chrisn/miniconda3", :executable "/home/chrisn/miniconda3/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/home/chrisn/miniconda3", :base-prefix "/home/chrisn/miniconda3", :libname "python3.9m", :base-exec-prefix "/home/chrisn/miniconda3", :python-home "/home/chrisn/miniconda3", :version [3 9 1], :platform "linux"}
Feb 10, 2022 7:09:57 AM clojure.tools.logging$eval5863$fn__5866 invoke
INFO: Prefixing java library path: /home/chrisn/miniconda3/lib
Feb 10, 2022 7:09:58 AM clojure.tools.logging$eval5863$fn__5866 invoke
INFO: Loading python library: python3.9
Feb 10, 2022 7:09:59 AM clojure.tools.logging$eval5863$fn__5866 invoke
INFO: Reference thread starting
So why didn't we see that logging information in your console? This is with no logging configured. We also didn't see a log4j message stating that console logging was disabled. Do you have some user-specific deps.edn configuration that is disabling all logging?
This is what happens if I try under JDK-17:
(base) chrisn@chrisn-lt3:~/dev/clj-python/libpython-clj$ java --version
openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-39)
OpenJDK 64-Bit Server VM (build 17.0.1+12-39, mixed mode, sharing)
(base) chrisn@chrisn-lt3:~/dev/clj-python/libpython-clj$ clj
Clojure 1.10.2
(require '[libpython-clj2.python :as py])
nil
(require '[libpython-clj2.require :refer [require-python])
Syntax error reading source at (REPL:2:59).
Unmatched delimiter: )
(require '[libpython-clj2.require :refer [require-python]])
Feb 10, 2022 7:13:20 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Detecting startup info
Feb 10, 2022 7:13:21 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Startup info {:lib-version "3.9", :java-library-path-addendum "/home/chrisn/miniconda3/lib", :exec-prefix "/home/chrisn/miniconda3", :executable "/home/chrisn/miniconda3/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/home/chrisn/miniconda3", :base-prefix "/home/chrisn/miniconda3", :libname "python3.9m", :base-exec-prefix "/home/chrisn/miniconda3", :python-home "/home/chrisn/miniconda3", :version [3 9 1], :platform "linux"}
Feb 10, 2022 7:13:21 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Prefixing java library path: /home/chrisn/miniconda3/lib
Feb 10, 2022 7:13:22 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Loading python library: python3.9
Feb 10, 2022 7:13:22 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Reference thread starting
Feb 10, 2022 7:13:22 AM clojure.tools.logging$eval659$fn__662 invoke
INFO: Unable to find direct buffer constructor -
falling back to jdk16 memory model.
Feb 10, 2022 7:13:22 AM clojure.tools.logging$eval659$fn__662 invoke
SEVERE: Exception while releasing object
Syntax error compiling at (tech/v3/datatype/ffi/nio_buf_mmodel.clj:1:1).
at clojure.lang.Compiler.load(Compiler.java:7652)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__6857.invoke(core.clj:6115)
at clojure.core$load.invokeStatic(core.clj:6114)
at clojure.core$load.doInvoke(core.clj:6098)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5897)
at clojure.core$load_one.invoke(core.clj:5892)
at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
at clojure.core$load_lib.invokeStatic(core.clj:5936)
at clojure.core$load_lib.doInvoke(core.clj:5917)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$load_libs.invokeStatic(core.clj:5974)
at clojure.core$load_libs.doInvoke(core.clj:5958)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$require.invokeStatic(core.clj:5996)
at clojure.core$require.doInvoke(core.clj:5996)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$serialized_require.invokeStatic(core.clj:6068)
at clojure.core$requiring_resolve.invokeStatic(core.clj:6077)
at clojure.core$requiring_resolve.invoke(core.clj:6071)
at tech.v3.datatype.nio_buffer$fn__8001.invokeStatic(nio_buffer.clj:104)
at tech.v3.datatype.nio_buffer$fn__8001.invoke(nio_buffer.clj:95)
at clojure.lang.Delay.deref(Delay.java:42)
at clojure.core$deref.invokeStatic(core.clj:2324)
at clojure.core$deref.invoke(core.clj:2310)
at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invokeStatic(nio_buffer.clj:113)
at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invoke(nio_buffer.clj:108)
at libpython_clj2.python.ffi$pystr__GT_str.invokeStatic(ffi.clj:785)
at libpython_clj2.python.ffi$pystr__GT_str.invoke(ffi.clj:773)
at libpython_clj2.python.ffi$pytype_name.invokeStatic(ffi.clj:793)
at libpython_clj2.python.ffi$pytype_name.invoke(ffi.clj:789)
at libpython_clj2.python.ffi$pyobject_type_kwd$reify__8624.apply(ffi.clj:809)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
at libpython_clj2.python.ffi$pyobject_type_kwd.invokeStatic(ffi.clj:805)
at libpython_clj2.python.ffi$pyobject_type_kwd.invoke(ffi.clj:802)
at libpython_clj2.python.ffi$simplify_or_track.invokeStatic(ffi.clj:949)
at libpython_clj2.python.ffi$simplify_or_track.invoke(ffi.clj:941)
at libpython_clj2.python.fn$call_py_fn.invokeStatic(fn.clj:197)
at libpython_clj2.python.fn$call_py_fn.invoke(fn.clj:177)
at libpython_clj2.python.fn$eval9449$fn__9450.invoke(fn.clj:203)
at libpython_clj2.python.protocols$eval8942$fn__8956$G__8931__8965.invoke(protocols.clj:67)
at libpython_clj2.python.fn$call.invokeStatic(fn.clj:211)
at libpython_clj2.python.fn$call.doInvoke(fn.clj:208)
at clojure.lang.RestFn.invoke(RestFn.java:460)
at libpython_clj2.python.class$create_class$fn__9686.invoke(class.clj:83)
at libpython_clj2.python.class$create_class.invokeStatic(class.clj:80)
at libpython_clj2.python.class$create_class.invoke(class.clj:72)
at libpython_clj2.python.io_redirect$fn__10474$fn__10475.invoke(io_redirect.clj:21)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1977)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1977)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at libpython_clj2.python.io_redirect$fn__10474.invokeStatic(io_redirect.clj:20)
at libpython_clj2.python.io_redirect$fn__10474.invoke(io_redirect.clj:20)
at clojure.lang.Delay.deref(Delay.java:42)
at clojure.core$deref.invokeStatic(core.clj:2324)
at clojure.core$deref.invoke(core.clj:2310)
at libpython_clj2.python.io_redirect$setup_std_writer$fn__10482.invoke(io_redirect.clj:46)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1977)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1977)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at libpython_clj2.python.io_redirect$setup_std_writer.invokeStatic(io_redirect.clj:44)
at libpython_clj2.python.io_redirect$setup_std_writer.invoke(io_redirect.clj:40)
at libpython_clj2.python.io_redirect$redirect_io_BANG_.invokeStatic(io_redirect.clj:54)
at libpython_clj2.python.io_redirect$redirect_io_BANG_.invoke(io_redirect.clj:52)
at libpython_clj2.python$initialize_BANG_$fn__10513.invoke(python.clj:107)
at libpython_clj2.python$initialize_BANG_.invokeStatic(python.clj:100)
at libpython_clj2.python$initialize_BANG_.doInvoke(python.clj:46)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at libpython_clj2.metadata$eval10798.invokeStatic(metadata.clj:17)
at libpython_clj2.metadata$eval10798.invoke(metadata.clj:17)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.load(Compiler.java:7640)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__6857.invoke(core.clj:6115)
at clojure.core$load.invokeStatic(core.clj:6114)
at clojure.core$load.doInvoke(core.clj:6098)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5897)
at clojure.core$load_one.invoke(core.clj:5892)
at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
at clojure.core$load_lib.invokeStatic(core.clj:5936)
at clojure.core$load_lib.doInvoke(core.clj:5917)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$load_libs.invokeStatic(core.clj:5974)
at clojure.core$load_libs.doInvoke(core.clj:5958)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$require.invokeStatic(core.clj:5996)
at clojure.core$require.doInvoke(core.clj:5996)
at clojure.lang.RestFn.invoke(RestFn.java:482)
at libpython_clj2.require$eval10782$loading__6738__auto____10783.invoke(require.clj:1)
at libpython_clj2.require$eval10782.invokeStatic(require.clj:1)
at libpython_clj2.require$eval10782.invoke(require.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.eval(Compiler.java:7170)
at clojure.lang.Compiler.load(Compiler.java:7640)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:372)
at clojure.lang.RT.load(RT.java:459)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__6857.invoke(core.clj:6115)
at clojure.core$load.invokeStatic(core.clj:6114)
at clojure.core$load.doInvoke(core.clj:6098)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5897)
at clojure.core$load_one.invoke(core.clj:5892)
at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
at clojure.core$load_lib.invokeStatic(core.clj:5936)
at clojure.core$load_lib.doInvoke(core.clj:5917)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$load_libs.invokeStatic(core.clj:5974)
at clojure.core$load_libs.doInvoke(core.clj:5958)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$require.invokeStatic(core.clj:5996)
at clojure.core$require.doInvoke(core.clj:5996)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval10776.invokeStatic(NO_SOURCE_FILE:1)
at user$eval10776.invoke(NO_SOURCE_FILE:1)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.eval(Compiler.java:7136)
at clojure.core$eval.invokeStatic(core.clj:3202)
at clojure.core$eval.invoke(core.clj:3198)
at clojure.main$repl$read_eval_print__9112$fn__9115.invoke(main.clj:437)
at clojure.main$repl$read_eval_print__9112.invoke(main.clj:437)
at clojure.main$repl$fn__9121.invoke(main.clj:458)
at clojure.main$repl.invokeStatic(main.clj:458)
at clojure.main$repl_opt.invokeStatic(main.clj:522)
at clojure.main$main.invokeStatic(main.clj:667)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
Caused by: java.lang.ClassNotFoundException: jdk.incubator.foreign.MemoryAddress
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at clojure.lang.RT.classForName(RT.java:2212)
at clojure.lang.RT.classForNameNonLoading(RT.java:2225)
at tech.v3.datatype.ffi.nio_buf_mmodel$eval13834$loading__6738__auto____13835.invoke(nio_buf_mmodel.clj:1)
at tech.v3.datatype.ffi.nio_buf_mmodel$eval13834.invokeStatic(nio_buf_mmodel.clj:1)
at tech.v3.datatype.ffi.nio_buf_mmodel$eval13834.invoke(nio_buf_mmodel.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.eval(Compiler.java:7170)
at clojure.lang.Compiler.load(Compiler.java:7640)
... 152 more
Syntax error (ClassNotFoundException) compiling at (tech/v3/datatype/ffi/nio_buf_mmodel.clj:1:1).
jdk.incubator.foreign.MemoryAddress
That last message indicates the real error under the covers. So again, the first thing I think would be a good idea would be to figure out why your logging is completely disabled and re-enable it.
logging information was shown when lanching repl with lein repl
nelson@MacBookPro libpython-clj-test % source ../cljvenv/bin/activate
(cljvenv) nelson@MacBookPro libpython-clj-test % lein repl
Feb 11, 2022 6:12:03 PM clojure.tools.logging$eval674$fn__677 invoke
INFO: Detecting startup info
Feb 11, 2022 6:12:03 PM clojure.tools.logging$eval674$fn__677 invoke
INFO: Startup info {:lib-version "3.9", :java-library-path-addendum "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib", :exec-prefix "/Users/Shared/Relocated_Items/Security/eUSB/sync/clojure/projects/cljvenv", :executable "/Users/Shared/Relocated_Items/Security/eUSB/sync/clojure/projects/cljvenv/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/Users/Shared/Relocated_Items/Security/eUSB/sync/clojure/projects/cljvenv", :base-prefix "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9", :libname "python3.9m", :base-exec-prefix "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9", :python-home "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9", :version [3 9 10], :platform "darwin"}
Feb 11, 2022 6:12:03 PM clojure.tools.logging$eval674$fn__677 invoke
INFO: Prefixing java library path: /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib
Feb 11, 2022 6:12:04 PM clojure.tools.logging$eval674$fn__677 invoke
INFO: Loading python library: /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by tech.v3.datatype.UnsafeUtil (file:/Users/nelson/.m2/repository/cnuernber/dtype-next/8.062/dtype-next-8.062.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of tech.v3.datatype.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Feb 11, 2022 6:12:04 PM clojure.tools.logging$eval674$fn__677 invoke
INFO: Reference thread starting
nREPL server started on port 55488 on host 127.0.0.1 - nrepl://127.0.0.1:55488
REPL-y 0.5.1, nREPL 0.8.3
Clojure 1.10.3
OpenJDK 64-Bit Server VM 11.0.14+9-jvmci-22.0-b05
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
libpython-clj-test.core=> (require '[libpython-clj2.python :as py])
nil
libpython-clj-test.core=> (require '[libpython-clj2.require :refer [require-python]])
nil
libpython-clj-test.core=> (def np (py/import-module "numpy"))
#'libpython-clj-test.core/np
libpython-clj-test.core=> (py/py. np linspace 2 3 :num 10)
[2. 2.11111111 2.22222222 2.33333333 2.44444444 2.55555556
2.66666667 2.77777778 2.88888889 3. ]
Closing this as I think this issue is resolved for now.
Ran into the same issue:
MacOS Monterey Version 12.6 java 17.0.2 2022-01-18 LTS
When importing
[libpython-clj2.require :refer [require-python]]
[libpython-clj2.python :refer [py. py.. py.-] :as py]
I would get
; Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:445).
; jdk.incubator.foreign.MemoryAddress
or
; Syntax error compiling at (libpython_clj2/require.clj:1:1).
; namespace 'libpython-clj2.metadata' not found
You have to put the following into your deps.edn
:aliases {:dev {:jvm-opts ["--add-modules" "jdk.incubator.foreign"
"--enable-native-access=ALL-UNNAMED"]}}
And then it should work
Platform : OSX 11.6.3 Python : 3.9.10 JVM : openjdk 17.0.2 2022-01-18 OpenJDK Runtime Environment GraalVM CE 22.0.0.2 (build 17.0.2+8-jvmci-22.0-b05) OpenJDK 64-Bit Server VM GraalVM CE 22.0.0.2 (build 17.0.2+8-jvmci-22.0-b05, mixed mode, sharing)
When execute
(require '[libpython-clj2.require :refer [require-python]])
in repl, encounter the following errorReproduce the error
create a testing project location with Lein:
lien new libpython-clj-test
Install the following to the project.clj :
libpython-clj-test.core=> (require '[libpython-clj2.python :as py :refer [py. py.. py.-]]) nil libpython-clj-test.core=> (py/initialize!) :already-initialized libpython-clj-test.core=> (require '[libpython-clj2.require :refer [require-python]]) Syntax error compiling at (libpython_clj2/require.clj:1:1). namespace 'libpython-clj2.metadata' not found