Python 3.9 Doesn't Load

cartesian-theatrics commented 4 years ago

Hello,

Minor issue, but thought I would note that Python 3.9 doesn't seem to work.

➜  Workspace git:(develop) ✗ nix-shell -p 'clojure' 'python39'       

[nix-shell:~/Workspace]$ clj -Sdeps '{:deps {clj-python/libpython-clj {:mvn/version "1.45"}}}'-SForce
[Rebel readline] Type :repl/help for online help info
user=> (ns facial-rec.face-feature
  #_=>   (:require [libpython-clj.require :refer [require-python]]
  #_=>             [libpython-clj.python :refer [py. py.. py.-] :as py]
  #_=>             [tech.v2.datatype :as dtype]))
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Detecting startup-info for Python executable: 
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Startup info detected:
{:lib-version "3.9",
 :java-library-path-addendum
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4/lib",
 :exec-prefix
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4",
 :executable
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4/bin/python3",
 :libnames ("python3.9m" "python3.9"),
 :prefix "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4",
 :base-prefix
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4",
 :base-exec-prefix
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4",
 :python-home
 "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4",
 :version [3 9 0],
 :platform "linux"}

Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Trying python library names ["python3.9m" "python3.9" "python3.7m" "python3.6m"]
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Setting java library path: /nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Reference thread starting
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Library python3.9 found at [:java-library-path "/nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4/lib/libpython3.9.so"]
Jun 26, 2020 10:43:54 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Library c found at [:system "c"]
Syntax error (UnsatisfiedLinkError) compiling at (libpython_clj/metadata.clj:13:1).
Error looking up function 'PyCFunction_New': /nix/store/a3fykl1fi65s93lrv9aprmzzlz2hj9h7-python3-3.9.0a4/lib/libpython3.9.so: undefined symbol: PyCFunction_New
facial-rec.face-feature=>

However, 3.8 works fine:

➜  Workspace git:(develop) ✗ nix-shell -p 'clojure' 'python38'

[nix-shell:~/Workspace]$ clj -Sdeps '{:deps {clj-python/libpython-clj {:mvn/version "1.45"}}}' 
[Rebel readline] Type :repl/help for online help info
user=> (ns facial-rec.face-feature
  #_=>   (:require [libpython-clj.require :refer [require-python]]
  #_=>             [libpython-clj.python :refer [py. py.. py.-] :as py]
  #_=>             [tech.v2.datatype :as dtype]))
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Executing python initialize with options:{:python-executable nil, :program-name nil, :python-home nil, :library-path nil}
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Detecting startup-info for Python executable: 
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Startup info detected:
{:lib-version "3.8",
 :java-library-path-addendum
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2/lib",
 :exec-prefix
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2",
 :executable
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2/bin/python3",
 :libnames ("python3.8m" "python3.8"),
 :prefix "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2",
 :base-prefix
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2",
 :base-exec-prefix
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2",
 :python-home
 "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2",
 :version [3 8 2],
 :platform "linux"}

Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Trying python library names ["python3.8m" "python3.8" "python3.7m" "python3.6m"]
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Setting java library path: /nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Reference thread starting
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Library python3.8 found at [:java-library-path "/nix/store/g9fcsl4d9vjyipv46la9wxvzi514r7ib-python3-3.8.2/lib/libpython3.8.so"]
Jun 26, 2020 10:46:05 PM clojure.tools.logging$eval6368$fn__6371 invoke
INFO: Library c found at [:system "c"]
nil
facial-rec.face-feature=>

cnuernber commented 4 years ago

Thanks for letting us know; looks like they removed a key function we use.

jjtolton commented 4 years ago

Jeeze we just got 3.8 working!! Python needs to slow down the release cadence haha

eoswald commented 4 years ago

That's strange, I get this same issue on Python 3.8.2:

PS C:\Users\PRISM-NODE-14\tests\libpython-test> cat .\src\libpython_test\core.clj
(ns libpython-test.core
  (:require [libpython-clj.python :as py]))

(defn -main
  [& args]
  (py/initialize! :python-executable "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python.exe"
                  :library-path "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll"))
PS C:\Users\PRISM-NODE-14\tests\libpython-test> C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python.exe -V                                                                                                             Python 3.8.2
PS C:\Users\PRISM-NODE-14\tests\libpython-test> lein run                                                                Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Executing python initialize with options:{:python-executable "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python.exe", :program-name nil, :python-home nil, :library-path "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll"}
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Detecting startup-info for Python executable: C:\Users\PRISM-NODE-14\AppData\Local\Programs\Python\Python38\python.exe
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Startup info detected:
{:lib-version "3.8",
 :java-library-path-addendum
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\lib",
 :exec-prefix
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38",
 :executable
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python.exe",
 :libnames
 ("C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll"
  "python3.8"),
 :prefix
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38",
 :base-prefix
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38",
 :base-exec-prefix
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38",
 :python-home
 "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38",
 :version [3 8 2],
 :platform "win32"}

Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Trying python library names ["C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll" "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll" "python3.8" "python3.7m" "python3.6m"]
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Setting java library path: C:\Users\PRISM-NODE-14\AppData\Local\Programs\Python\Python38\lib:C:\jdk-13.0.2\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\Git\cmd;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;D:\Program Files\PuTTY\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\PRISM-NODE-14\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\php;C:\jdk-13.0.2\bin;C:\bin;C:\Users\PRISM-NODE-14\AppData\Local\Microsoft\WindowsApps;C:\Users\PRISM-NODE-14\AppData\Local\Programs\Python\Python37-32;C:\Hugo\bin;C:\Users\PRISM-NODE-14\AppData\Local\Microsoft\WindowsApps;C:\Users\PRISM-NODE-14\AppData\Local\atom\bin;C:\Users\PRISM-NODE-14\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Users\PRISM-NODE-14\AppData\Local\Programs\Microsoft VS Code\bin;;.
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Reference thread starting
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Library C:\Users\PRISM-NODE-14\AppData\Local\Programs\Python\Python38\python38.dll found at [:system "C:\\Users\\PRISM-NODE-14\\AppData\\Local\\Programs\\Python\\Python38\\python38.dll"]
Sep 01, 2020 9:52:55 PM clojure.tools.logging$eval513$fn__516 invoke
INFO: Library msvcrt found at [:system "msvcrt"]
Exception in thread "main" Syntax error compiling at (C:\Users\PRISM-NODE-14\AppData\Local\Temp\form-init7457453783192874858.clj:1:116).
        at clojure.lang.Compiler.load(Compiler.java:7647)
        at clojure.lang.Compiler.loadFile(Compiler.java:7573)
        at clojure.main$load_script.invokeStatic(main.clj:452)
        at clojure.main$init_opt.invokeStatic(main.clj:454)
        at clojure.main$init_opt.invoke(main.clj:454)
        at clojure.main$initialize.invokeStatic(main.clj:485)
        at clojure.main$null_opt.invokeStatic(main.clj:519)
        at clojure.main$null_opt.invoke(main.clj:516)
        at clojure.main$main.invokeStatic(main.clj:598)
        at clojure.main$main.doInvoke(main.clj:561)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.main.main(main.java:37)
Caused by: java.lang.UnsatisfiedLinkError: Error looking up function 'PyCFunction_New': The specified procedure could not be found.

        at com.sun.jna.Function.<init>(Function.java:251)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:594)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:570)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:521)
        at tech.jna.base$do_find_function.invokeStatic(base.clj:213)
        at tech.jna.base$do_find_function.invoke(base.clj:212)
        at tech.jna.base$find_function.invokeStatic(base.clj:218)
        at tech.jna.base$find_function.invoke(base.clj:216)
        at tech.jna$find_function.invokeStatic(jna.clj:92)
        at tech.jna$find_function.invoke(jna.clj:90)
        at libpython_clj.jna.concrete.cfunction$PyCFunction_New.invokeStatic(cfunction.clj:54)
        at libpython_clj.jna.concrete.cfunction$PyCFunction_New.invoke(cfunction.clj:54)
        at libpython_clj.python.object$cfunc_impl__GT_pyobject.invokeStatic(object.clj:447)
        at libpython_clj.python.object$cfunc_impl__GT_pyobject.invoke(object.clj:437)
        at libpython_clj.python.object$make_tuple_fn.invokeStatic(object.clj:518)
        at libpython_clj.python.object$make_tuple_fn.doInvoke(object.clj:485)
        at clojure.lang.RestFn.invoke(RestFn.java:410)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:667)
        at clojure.core$apply.invoke(core.clj:660)
        at libpython_clj.python.object$__GT_py_fn.invokeStatic(object.clj:536)
        at libpython_clj.python.object$__GT_py_fn.invoke(object.clj:521)
        at libpython_clj.python.object$eval30325$fn__30326.invoke(object.clj:670)
        at libpython_clj.python.protocols$eval25048$fn__25049$G__25039__25056.invoke(protocols.clj:34)
        at libpython_clj.python.object$__GT_python.invokeStatic(object.clj:220)
        at libpython_clj.python.object$__GT_python.doInvoke(object.clj:215)
        at clojure.lang.RestFn.invoke(RestFn.java:410)
        at libpython_clj.python.interop$create_var_writer.invokeStatic(interop.clj:422)
        at libpython_clj.python.interop$create_var_writer.invoke(interop.clj:416)
        at libpython_clj.python.interop$get_or_create_var_writer.invokeStatic(interop.clj:436)
        at libpython_clj.python.interop$get_or_create_var_writer.invoke(interop.clj:431)
        at libpython_clj.python.interop$setup_std_writer.invokeStatic(interop.clj:443)
        at libpython_clj.python.interop$setup_std_writer.invoke(interop.clj:439)
        at libpython_clj.python$initialize_BANG_.invokeStatic(python.clj:249)
        at libpython_clj.python$initialize_BANG_.doInvoke(python.clj:229)
        at clojure.lang.RestFn.invoke(RestFn.java:457)
        at libpython_test.core$_main.invokeStatic(core.clj:6)
        at libpython_test.core$_main.doInvoke(core.clj:4)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.Var.invoke(Var.java:380)
        at user$eval140.invokeStatic(form-init7457453783192874858.clj:1)
        at user$eval140.invoke(form-init7457453783192874858.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7176)
        at clojure.lang.Compiler.eval(Compiler.java:7166)
        at clojure.lang.Compiler.load(Compiler.java:7635)
        ... 12 more

cnuernber commented 4 years ago

We need that function to work with the current design; it is how we register clojure functions as python functions.

It would be helpful if someone could research what happened to that function.

sogaiu commented 4 years ago

Is it defined here?

3.7 branch: https://github.com/python/cpython/blob/3.7/Objects/methodobject.c#L21-L25

3.8 branch: https://github.com/python/cpython/blob/3.8/Objects/methodobject.c#L33-L37

3.9 branch: https://github.com/python/cpython/blob/3.9/Objects/methodobject.c#L31-L35

cnuernber commented 4 years ago

Apparently but if it isn't exported from the shared library then we can't get to the symbol. So there must be an export step where this got removed.

Try searching shared libraries for symbols; that is the real test here. On Linux I think I use nm -d. In any case jna can't find the symbol and that is what is causing the failure.

sogaiu commented 4 years ago

Does this help?

$ nm -D libpython3.9.so.1.0 | grep PyCFunction_New
00000000000e0b40 T PyCFunction_NewEx

That was the only hit.

In contrast, for 3.8 I got:

$ nm -D libpython3.8.so.1.0 | grep PyCFunction_New
00000000000c9680 T PyCFunction_New
00000000000c94b0 T PyCFunction_NewEx

cnuernber commented 4 years ago

Ah, so perhaps we can use PyCFunction_NewEx....yes that helps.

Does it have a crazy function signature?

sogaiu commented 4 years ago

Looks like PyCFunction_New just calls PyCFunction_NewEx with the last argument as NULL?

https://github.com/python/cpython/blob/3.9/Objects/methodobject.c#L31-L42

PyObject *
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
    return PyCFunction_NewEx(ml, self, NULL);
}

PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
{
    return PyCMethod_New(ml, self, module, NULL);
}

cnuernber commented 4 years ago

Awesome, thank you, that looks to be a way forward. We can test that one out fairly easily and see what happens. I was pretty concerned that we would need to do something like build a shim for 3.9+ so this actually really eases my mind a bit.

sogaiu commented 4 years ago

Regarding the report in: https://github.com/clj-python/libpython-clj/issues/105#issuecomment-685235010

That looks like a Windows environment (the report of this happening on 3.8.2). Perhaps a platform difference?

cnuernber commented 4 years ago

Potential fix in 1.46. Thanks to everyone who chimed in here and especially @sogaiu - That changed work for me and appears to be backwards compatible.

Tentatively closing!

clj-python / libpython-clj

Python 3.9 Doesn't Load #105