Closed metasoarous closed 1 year ago
Pinging @cnuernber on this one.
So JDK-18, 19 aren't at this time supported as they both have incompatible changes to the pathway from an integer to a direct buffer.
Specifically 16, 17, 18, 19 all contain incompatible changes there so I have been avoiding upgrading dtype-next to those versions. At this time only JDK-17 is supported beyond JDK-11.
Got it :+1: Thanks @cnuernber!
I don't have a strong reason for using any particular JDK version, so happy to stick with JDK 17 for now. It's too bad that we're having to fuss over platform "changes" though. In theory, if someone were motivated, could newer versions be supported?
Thanks!
Of course. Its just work - scanning the api changes and fixing dtype-next.
I switched to JDK 17, and am seeing the same behavior as before.
For context, I'm pretty sure libpythonclj is finding the correct python installation. And functions like py/->python
works. But others do not (such as py/as-python
and py/->jvm
).
I have a pretty minimal reproduction of this in a Docker container at https://github.com/metasoarous/libpython-require-error. Please let me know if you see anything obviously wrong there, or have hints about what I should be looking for.
Thanks again
@metasoarous I see your issue -- and raise you an issue! https://github.com/metasoarous/libpython-require-error/issues/1
Thanks @jjtolton; That should be resolved now. Please let me know if it gets you to up the issue stack :-)
Much appreciated. I'll dive in shortly!
One thing I want to note is that to my knowledge :jvm-opts
only works in an alias. A top-level :jvm-opts
is silently ignored. So I think the example deps.edn needs an alias with it in it and the dockerfile's cmd needs to be updated to include that alias.
As noted in the other issue if you don't see exactly this warning:
WARNING: Using incubator modules: jdk.incubator.foreign
Then you have not loaded the module.
@metasoarous , I reconfigured your deps.edn
per @cnuernber 's instructions.
Current file:
{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.11.1"}
techascent/tech.ml.dataset {:mvn/version "6.093"}
;techascent/tech.ml.dataset {:mvn/version "6.086"
;:exclusions [org.slf4j/slf4j-api]}
org.bytedeco/mkl-platform-redist {:mvn/version "2020.1-1.5.3"}
clj-python/libpython-clj {:mvn/version "2.018"}
cider/cider-nrepl {:mvn/version "0.28.5"}
com.taoensso/timbre {:mvn/version "5.2.1"}
ch.qos.logback/logback-classic {:mvn/version "1.1.3"}}
:aliases {:opts {:jvm-opts ["--add-modules" "jdk.incubator.foreign"
"--enable-native-access=ALL-UNNAMED"]}}}
I now get a different error, but still not working. Note:
root@802025a3b951:/app# java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
New error:
root@802025a3b951:/app# clj -A:opts -m libpython-test.test
WARNING: Implicit use of clojure.main with options is deprecated, use -M
WARNING: Using incubator modules: jdk.incubator.foreign
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/infinite?
WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/abs
WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/infinite?
20:15:39.461 [main] INFO libpython-clj2.python.info - Detecting startup info
20:15:39.502 [main] INFO libpython-clj2.python - Startup info {:lib-version "3.9", :java-library-path-addendum "/usr/lib", :exec-prefix "/usr", :executable "/usr/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/usr", :base-prefix "/usr", :libname "python3.9m", :base-exec-prefix "/usr", :python-home "/usr", :version [3 9 2], :platform "linux"}
20:15:39.502 [main] INFO libpython-clj2.python - Prefixing java library path: /usr/lib
20:15:39.577 [main] INFO libpython-clj2.python - Loading python library: python3.9
20:15:40.095 [main] DEBUG libpython-clj2.python.ffi - Initializing Python C Layer
20:15:40.098 [tech.resource.gc ref thread] INFO tech.v3.resource.gc - Reference thread starting
20:15:40.100 [main] DEBUG libpython-clj2.python.ffi - Python Home: /usr
20:15:40.124 [main] INFO tech.v3.datatype.nio-buffer - Unable to find direct buffer constructor -
falling back to jdk16 memory model.
:OK
Execution error (NullPointerException) at clojure.main/main (main.java:40).
Cannot invoke "clojure.lang.IFn.applyTo(clojure.lang.ISeq)" because "f" is null
Full report at:
/tmp/clojure-7296416279702110182.edn
Hopefully this is a more tractable error?
I think to rule out installation issues, I will try to merge the example Dockerfile with the libpython-clj example Dockerfile to try to rule out installation/version issues.
@metasoarous not sure if this solves the initial problem or not, but I put in a PR to your repo so that at least you have a minimal working setup: https://github.com/metasoarous/libpython-require-error/pull/2
@cnuernber D'oh! I did not in fact realize that this is a constraint with :jvm-opts
. Not clear to me why that's not possible but :shrug: Thanks for pointing out.
@jjtolton Thank you for digging in! As mentioned in #213, I was able to get baseline functionality working by starting FROM clojure:openjdk-8-tools-deps
, which I think is the same version used in the libpython-clj example Dockerfile. I may be fine continuing to use that, which would at least remove any immediate pressure for me, but it would still be nice to sort out connecting to jdk-17 if possible. (PS I had jdk-11 working at one point, but it stopped building some of the python pieces, presumably because the dockerfiles on dockerhub were updated with some weird environmental artifact :shrug:)
Thanks again!
No problem, @metasoarous. Will tinker around later and see if I can a Dockerfile for JDK-17 going. Hopefully this gives you enough for now.
First off, just wanted to confirm @cnuernber's suspicion that I wasn't using :jdk-opts
properly. Now that I've done that, at least jdk-17 works. I'm happy to stick with that for a good while.
It's up to you whether you want to leave this issue open to track jdk-19 support, in case someone is inspired to work on it. I think it would be great for beginner friendliness to the ecosystem if folks aren't having to worry about JDK version so much. Trying to at least support the LTS versions is probably a pretty tenable path forward if you want to keep things simple.
Regardless, I think the README could be made a bit clearer regarding which versions are supported. I'm happy to submit an issue &/or PR for that if you agree.
Thanks again
Great point. Documentation is key for this experience! If it is fresh in your mind about where you would expect to see that documented, I would encourage you to submit a PR for it!
On Sun, Sep 4, 2022 at 1:19 AM Christopher Small @.***> wrote:
First off, just wanted to confirm @cnuernber https://github.com/cnuernber's suspicion that I wasn't using :jdk-opts properly. Now that I've done that, at least jdk-17 works. I'm happy to stick with that for a good while.
It's up to you whether you want to leave this issue open to track jdk-19 support, in case someone is inspired to work on it. I think it would be great for beginner friendliness to the ecosystem if folks aren't having to worry about JDK version so much. Trying to at least support the LTS versions is probably a pretty tenable path forward if you want to keep things simple.
Regardless, I think the README could be made a bit clearer regarding which versions are supported. I'm happy to submit an issue &/or PR for that if you agree.
Thanks again
— Reply to this email directly, view it on GitHub https://github.com/clj-python/libpython-clj/issues/212#issuecomment-1236262207, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPJX47YCP67467YQ7BPXFTV4QWMLANCNFSM6AAAAAAQCVTTMY . You are receiving this because you were mentioned.Message ID: @.***>
@metasoarous set a PR for a working reference copy with JDK-17
2.023 and dtype-next version 10.000-beta-18 FIX THIS ISSUE :-)
@cnuernber AMAZING! Thank you kindly for all your work!
Hello, just a note for people reaching for a problem similar to mine and end up here, in case it saves someone's time: Java 20 no longer has idk.incubator.foreign as a module, so your programs might have broken if you autoupdated your openjdk version (brew did this to me automatically luckily there's still openjdk@17).
I'm trying out libpython-clj 2.018 on JDK 19, and I'm getting the following error, both the first time I try to initialize, and when attempting to require
libpython-clj2.require
:I saw issue #180, which suggested adding
:jvm-opts ["--add-modules" "jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"]
to mydeps.edn
for JVM versions 17 and up. I may tinker around with the version a bit, but it will require rebuilding docker images, so it would be helpful to know if anyone has any experience using libpython-clj 2 with JDK 19.Thanks