search

clj-python / libpython-clj

Python bindings for Clojure
Eclipse Public License 2.0
1.08k stars 68 forks source link

Add support for python 3.8 #98

Closed kopos closed 4 years ago

kopos commented 4 years ago

In src/libpython_clj/jna/base.clj, the dynamic vars *python-library-names* is set to ["python3.7m" "python3.6m"].

Please add python3.8m to this vec.

cassc commented 4 years ago

Hi, you can specify paths for your python version in py/initialize!: 

(defn init-python! []
  (py/initialize! :python-executable "/usr/bin/python3.8"
                  :library-path "/usr/lib/libpython3.8.so"))

(defonce python-initer (delay (init-python!)))

@python-initer
kopos commented 4 years ago

I've followed your suggestion and it works fine. Thank you.

However, without the fix, the logs (line 3) seem to indicate that libpython is automatically detecting the lib-version as 3.8. But fails loading...

09:46:06.720 [main] INFO  libpython-clj.python.logging - Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}

09:46:06.721 [main] INFO  libpython-clj.python.logging - Detecting startup-info for Python executable: 

09:46:06.746 [main] INFO  libpython-clj.python.interpreter - Startup info detected: {:python-home "/usr", :lib-version "3.8", :libname "python3.8m", :java-library-path-addendum "/usr/lib"}

09:46:06.749 [tech.resource.gc ref thread] INFO  tech.resource.gc - Reference thread starting
Syntax error (ExceptionInfo) compiling at (libpython_clj/metadata.clj:13:1).
Failed to load library
kopos commented 4 years ago

Thanks @cassc.

Setting the python-executable and library path while explicitly initializing py works fine!

(ns some.ns
  (:require
    [environ.core :refer [env]]
    [clojure.tools.logging :as log]
    [clojure.math.combinatorics :as c]))

(defn init-python! []
  (if (some? (:python-executable env))
    (py/initialize!
      :python-executable (:python-executable env)
      :library-path (:library-path env))))

(defonce python-initer (delay (init-python!)))

(deref python-initer)

(require '[libpython-clj.require :refer [require-python]])

(require-python '[numpy :as np])
(require-python '[numpy.ma :as ma])
(require-python '[scipy.ndimage :as ndimage])

...
kopos commented 4 years ago

I've followed your suggestion and it works fine. Thank you.

However, without the fix, the logs (line 3) seem to indicate that libpython is automatically detecting the lib-version as 3.8. But fails loading...

09:46:06.720 [main] INFO  libpython-clj.python.logging - Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}

09:46:06.721 [main] INFO  libpython-clj.python.logging - Detecting startup-info for Python executable: 

09:46:06.746 [main] INFO  libpython-clj.python.interpreter - Startup info detected: {:python-home "/usr", :lib-version "3.8", :libname "python3.8m", :java-library-path-addendum "/usr/lib"}

09:46:06.749 [tech.resource.gc ref thread] INFO  tech.resource.gc - Reference thread starting
Syntax error (ExceptionInfo) compiling at (libpython_clj/metadata.clj:13:1).
Failed to load library

Hi @cassc

‘Startup info detected: {:python-home "/usr", :lib-version "3.8", :libname "python3.8m", :java-library-path-addendum "/usr/lib"}‘

Is this not an issue? The default initialisation seems to detect ‘3.8’ correctly but fails loading it...

jjtolton commented 4 years ago

Hiya @kopos ! I can't tell if your issue is resolved or not?

jjtolton commented 4 years ago

@kopos if you still can't get it working I have some time this weekend and I'll try to dig a little deeper

kopos commented 4 years ago

Hiya @kopos ! I can't tell if your issue is resolved or not?

Hi @jjtolton ,

Thank you for reviewing this.

  1. The fix by explicitly setting the python-exectuable and library-path works. As shown here https://github.com/clj-python/libpython-clj/issues/98#issuecomment-627132735. In as much as getting libpython-cli to work with python 3.8, the issue is resolved.

  2. The other issue, however, is that from the logs here https://github.com/clj-python/libpython-clj/issues/98#issuecomment-627132541, it seems that libpython-clj.python.interpreter is already correctly determining the python version as 3.8 etc without even setting the python-exectuable and library-path explicitly. So seems to me, that something is lost between libpython-clj.python.interpreter and libpython-clj.metadata

Is there something I can do to help here?

jjtolton commented 4 years ago

I just want to clarify -- are you able to get it working or does this still crash for you?

jjtolton commented 4 years ago

I haven't completed investigation but preliminary poking around shows that libpython3.8m.so isn't getting installed in the same place as libpython3.7m.so. In fact ... I haven't found it at all yet. I have found /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0. @cnuernber can you confirm if this does the same thing or not? If so then we just need to tune the regex here.

kopos commented 4 years ago

I just want to clarify -- are you able to get it working or does this still crash for you?

Able to get it working by setting the config vars explicitly

kopos commented 4 years ago

I haven't completed investigation but preliminary poking around shows that libpython3.8m.so isn't getting installed in the same place as libpython3.7m.so. In fact ... I haven't found it at all yet. I have found /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0. @cnuernber can you confirm if this does the same thing or not? If so then we just need to tune the regex here.

From what you've pointed out @jjtolton, as I can see the problem seems to be with python-library-regex fn in libpython-clj.python.interpreter ns

(require '[libpython-clj.python.interpreter :as lc])

(let [executable "python3.8"
       system-info (lc/python-system-info executable)
       pyregex (lc/python-library-regex system-info)]
   (println pyregex)
   (lc/python-library-paths system-info pyregex)))
#"libpython3.8m.so$"
[]

By changing the pyregex to accommodate the optional m #"libpython3.8m?.so$", the paths are being returned correctly.

rumbo.main=> (lc/python-library-paths (lc/python-system-info "python3.8") #"libpython3.8m?.so$")
["/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so" "/usr/lib/x86_64-linux-gnu/libpython3.8.so"]
cnuernber commented 4 years ago

@kopos - This is great, will update and produce a new version of the library.

cnuernber commented 4 years ago

Release 1.44 is up.

kopos commented 4 years ago

Thank you @jjtolton @cassc @cnuernber 🙏

It was completely to @jjtolton ‘s credit to picking this up! 🙏

cnuernber commented 4 years ago

@kopos - Please try it out and see. I added python3.8 to the list of executables to try in order to build the python executable system information and then it will now look for 3.8.[so,dylib] and 3.8m.[so,dylib].

The docker container I used to test all of this is under the main repository dockerfiles and the bash script I use to start it is also under scripts/run-py38-docker.

kopos commented 4 years ago

1.44 is working like a charm! Thank you @cnuernber