Beta 2.000 not loading: "Failed to find a valid python library!"

[mtruyens]

I just tried version 2.000, with the Spacy example https://github.com/gigasquid/libpython-clj-examples/blob/master/src/gigasquid/spacy.clj (but modifying the :require imports to v2).

However, I get the following error:

"Failed to find a valid python library!"

It worked perfectly on 1.46.

I'm on MacOS Big Sur, Python 3.9.1.

[cnuernber]

Could you paste all of the log output from version 1.46 and from version 2.000?

Where did 1.46 find the shared library?

[mtruyens]

Thank you for the swift reply!

Sorry for my ignorance, but how do I paste the log output? The only output that 2.000 gives me upon trying to load a namespace that contains

[libpython-clj2.require :refer [require-python]]
[libpython-clj2.python :as py :refer [py. py.. py.-]]

is the following:

Syntax error compiling at (libpython_clj2/metadata.clj:17:1).
Failed to find a valid python library!

On the contrary, 1.46 just says nothing...

[cnuernber]

There should be substantial logging. For instance, on my system -

user> (require '[libpython-clj2.python :as py])
nil
user> (py/initialize!)
Aug 30, 2021 11:10:09 AM clojure.tools.logging$eval7596$fn__7599 invoke
INFO: Detecting startup info
Aug 30, 2021 11:10:09 AM clojure.tools.logging$eval7596$fn__7599 invoke
INFO: Startup info {:lib-version "3.9", :java-library-path-addendum "/home/chrisn/miniconda3/lib", :exec-prefix "/home/chrisn/miniconda3", :executable "/home/chrisn/miniconda3/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/home/chrisn/miniconda3", :base-prefix "/home/chrisn/miniconda3", :libname "python3.9m", :base-exec-prefix "/home/chrisn/miniconda3", :python-home "/home/chrisn/miniconda3", :version [3 9 1], :platform "linux"}
Aug 30, 2021 11:10:09 AM clojure.tools.logging$eval7596$fn__7599 invoke
INFO: Prefixing java library path: /home/chrisn/miniconda3/lib
Aug 30, 2021 11:10:10 AM clojure.tools.logging$eval7596$fn__7599 invoke
INFO: Loading python library: python3.9
Aug 30, 2021 11:10:10 AM clojure.tools.logging$eval7596$fn__7599 invoke
INFO: Reference thread starting
:ok

[cnuernber]

That Startup info line is the key one and both versions should produce it. If not then my guess is that the logging system is disabled and someone included slf4j-api but no slf4j-compatible backend.

[mtruyens]

You were right about the missing slf4j!

Output from 1.4:

sx.clj.db.core=> 2021-08-30 19:44:30 INFO  logging:326 - Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
2021-08-30 19:44:30 INFO  logging:326 - Detecting startup-info for Python executable: 
2021-08-30 19:44:30 INFO  interpreter:326 - Startup info detected:
{:lib-version "3.9",
 :java-library-path-addendum
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib",
 :exec-prefix
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9",
 :executable "/usr/local/opt/python@3.9/bin/python3.9",
 :libnames ("python3.9m" "python3.9"),
 :prefix
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9",
 :base-prefix
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9",
 :base-exec-prefix
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9",
 :python-home
 "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9",
 :version [3 9 1],
 :platform "darwin"}

2021-08-30 19:44:30 INFO  interpreter:326 - Trying python library names ["python3.9m" "python3.9" "python3.7m" "python3.6m"]
2021-08-30 19:44:30 INFO  interpreter:326 - Setting java library path: /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib:/Users/mtruyens/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2021-08-30 19:44:30 INFO  gc:326 - Reference thread starting
2021-08-30 19:44:30 INFO  base:326 - Library python3.9 found at [:java-library-path "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/libpython3.9.dylib"]
2021-08-30 19:44:30 INFO  base:326 - Library c found at [:system "c"]

And from 2.000:

sx.clj.db.core=> 2021-08-30 19:45:28 INFO  info:286 - Detecting startup info
2021-08-30 19:45:28 INFO  python:286 - Startup info {:lib-version "3.9", :java-library-path-addendum "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib", :exec-prefix "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9", :executable "/usr/local/opt/python@3.9/bin/python3.9", :libnames ("python3.9m" "python3.9"), :prefix "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9", :base-prefix "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9", :libname "python3.9m", :base-exec-prefix "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9", :python-home "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9", :version [3 9 1], :platform "darwin"}
2021-08-30 19:45:28 INFO  python:286 - Prefixing java library path: /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib
2021-08-30 19:45:28 ERROR ffi:286 - Failed to find a suitable FFI implementation.
Attempted both :jdk and :jna -- call set-ffi-impl! from the repl to see specific failure.
Syntax error compiling at (insn/core.clj:1:1).
    at clojure.core$throw_if.invokeStatic(core.clj:5856)
    at clojure.core$load_lib.invokeStatic(core.clj:5942)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
        ....

[cnuernber]

What is happening here is that insn is failing to initialize. Specifically the version of org.ow2.asm/asm in your dependency chain is less than 7.0.

Try this in your deps.edn:

org.ow2.asm/asm                  {:mvn/version "9.0"}

[mtruyens]

Thanks, it is now working!

I may have missed something, but perhaps it is a good idea to explicitly warn users of this library to also include the following JDK params:

--add-modules jdk.incubator.foreign
-Dforeign.restricted=permit

This is documented in the Dtype-next project file, but it took me quite a while (and several error messages during startup) do figure this out.

Also, do I understand correctly that JDK-15 is simply not supported, so that either JDK-14 or 16 should be used?

[mtruyens]

But it any case thanks a million for the help, and this wonderful project. It's really wonderful how I can now use Spacy without awkward separate projects.

[cnuernber]

SpacyCpp is what kicked this entire thing off :-).

I agree, I should mention exactly those caveats on the docs for jdk-16. I think jdk-15 works, it really depends on exactly when they removed some of the unsafe stuff. The thing is a lot of this stuff is moving so fast that I get focused on something else.

Have you seen avclj?

In any case, it is great to hear this worked for you. Let's leave this issue open until I (or someone) does a doc pass on jdk-16+ support.

[cnuernber]

Updating this issue!!

We no longer support jdk-16 because jdk-17 introduced incompatible changes in the foreign incubator module. So we now support jdk-8, jdk-11, and jdk-17 with the readme updated to show how to use jdk-17 as suggested in this issue. So with that I think it is safe to close this issue :-).