Unable to load any external modules

[Camsbury]

In the past I had used the following to initialize:

(let [python-path (env :python-path)
      full-version (env :python-version)
      python-version (->> (index-of full-version ".")
                          inc
                          (index-of full-version ".")
                          (subs full-version 0))]
  (py/initialize!
   :python-executable (str python-path "/bin/python" python-version)
   :library-path (str python-path "/lib/libpython" python-version "m.dylib")))

However, I see on linux I'm supposed to use libpython3.Xm.so. In python 3.9, installed via nix on NixOS, it seems that the m is gone entirely, and I'm left with libpython3.9.so, which doesn't seem to work properly when I set it in my library-path.

Updated the above to:

(let [python-path (env :python-path)
      full-version (env :python-version)
      python-version (->> (index-of full-version ".")
                          inc
                          (index-of full-version ".")
                          (subs full-version 0))]
  (py/initialize!
   :python-executable (str python-path "/bin/python" python-version)
   :library-path (str python-path "/lib/libpython" python-version ".so")))

python-path - "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env" python-version - "3.9" if I do ls $PYTHON_PATH/lib I get the following:

libpython3.9.so -> /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/libpython3.9.so
libpython3.9.so.1.0 -> /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/libpython3.9.so.1.0
libpython3.so -> /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/libpython3.so
pkgconfig -> /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/pkgconfig
python3.9

The symlinks are just pointing to the *1.0 path under different names. Modules are available just fine if I run $PYTHON_PATH/bin/python.

[Camsbury]

Also tried directly pasting in the path to the eventual symlink target to the same results.

[cnuernber]

I am guessing that you mean 'import xyz' works in python and (import-module "xyz") does not work when libpython does eventually load your .so?

What is the output of the logging system when you attempt to load using your custom library path?

[cnuernber]

Specifically what is the output of (libpython-clj2.python.info/detect-startup-info) and does it make sense considering what else you know about your module setup?

The way that python finds modules has to do with exactly where python-home points to. So in addition to loading the appropriate shared library we need to setup the python system variables correctly and I am guessing something is off there.

[Camsbury]

I am guessing that you mean 'import xyz' works in python and (import-module "xyz") does not work when libpython does eventually load your .so?

I'm sorry, I realize now that was pretty vague. But you are right. I'll grab that info for you now.

[Camsbury]

initialize! just returns :ok, and (libpython-clj2.python.info/detect-startup-info) returns

{:lib-version "3.9",
 :java-library-path-addendum "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib",
 :exec-prefix "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env",
 :executable "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env/bin/python3.9",
 :libnames ("python3.9m" "python3.9"),
 :prefix "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env",
 :base-prefix "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :libname "python3.9m",
 :base-exec-prefix "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :python-home "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :version [3 9 10],
 :platform "linux"}

[Camsbury]

Maybe :libname should be "python3.9"?

[cnuernber]

It will try every member of libnames until it finds something it can load: https://github.com/clj-python/libpython-clj/blob/master/src/libpython_clj2/python.clj#L82

[Camsbury]

Logs happening if I defer requiring and running anything until after the repl starts up: (using CIDER)

2022-03-17T13:49:34.026Z ... INFO [libpython-clj2.python.info:326] - Detecting startup info for Python executable /nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env/bin/python3.9
2022-03-17T13:49:34.077Z ... INFO [libpython-clj2.python:326] - Startup info {:lib-version "3.9", :java-library-path-addendum "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib", :exec-prefix "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env", :executable "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env/bin/python3.9", :libnames ("python3.9m" "python3.9"), :prefix "/nix/store/9gpq185njp8m9hw0xz81yrykxd9qiy82-python3-3.9.10-env", :base-prefix "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10", :libname "python3.9m", :base-exec-prefix "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10", :python-home "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10", :version [3 9 10], :platform "linux"}
2022-03-17T13:49:34.078Z ... INFO [libpython-clj2.python:326] - Prefixing java library path: /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib
2022-03-17T13:49:34.544Z ... INFO [libpython-clj2.python:326] - Loading python library: /nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/libpython3.9.so
2022-03-17T13:49:34.630Z ... INFO [tech.v3.resource.gc:326] - Reference thread starting
Syntax error compiling at (src/ml/core.clj:6:1).
ModuleNotFoundError: No module named 'numpy'

And still (libpython-clj2.python.info/detect-startup-info) returns the same.

[Camsbury]

  (:require [base]
            [libpython-clj2.python :refer [py. py.. py.-] :as py]
            [libpython-clj2.require :refer [require-python]]))

(require-python '[numpy  :as np]
                '[pandas :as pd]
                '[kaggle.api :as kg])

The core ns for reference. (base holding the init code)

[cnuernber]

1. A couple things - numpy itself needs a c++ shared library to load, at least one depending on how it is built but usually you see this output in the error message.
2. python-home is also something you can pass in to override the detected python-home but something is missing here.
3. Do you know concretely where the numpy module is installed?

Another pathway is to use the embedded pathway to launch a clojure repl from python. You can still use libpython-clj's various functions and such.

[Camsbury]

Let's use requests instead as an example: /nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env/lib/python3.9/site-packages/requests/__init__.py (In this build PYTHON_PATH is /nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env)

[Camsbury]

Okay I realized I should have been running this:

  (clojure.pprint/pprint
   (let [python-path (env :python-path)
         full-version (env :python-version)
         python-version (->> (index-of full-version ".")
                             inc
                             (index-of full-version ".")
                             (subs full-version 0))]
     (libpython-clj2.python.info/detect-startup-info
      {:python-executable (str python-path "/bin/python" python-version)
       :library-path (str python-path "/lib/libpython" python-version ".so")})))

Which yields:

{:lib-version "3.9",
 :java-library-path-addendum
 "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib",
 :exec-prefix
 "/nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env",
 :executable
 "/nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env/bin/python3.9",
 :libnames
 ("/nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env/lib/libpython3.9.so"
  "python3.9"),
 :prefix
 "/nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env",
 :base-prefix
 "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :libname
 "/nix/store/cigj30pca7p38ipgl6lm12l35vx9jzq0-python3-3.9.10-env/lib/libpython3.9.so",
 :base-exec-prefix
 "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :python-home
 "/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10",
 :version [3 9 10],
 :platform "linux"}

[Camsbury]

Okay if I run the python in python-home above I have nothing in the way of libraries. Setting that explicitly, only base-exec-prefix and base-prefix remain pointing to the other path. Going to try to align these and see what happens.

[Camsbury]

That seems to fix it! python-home derived values are relying on the libs to be in that same path. In the past the two values I provided were enough, but this is still cleaner. (py/initialize! :python-home (env :python-path))

[Camsbury]

Also thanks for all the great work with the repo!

[cnuernber]

That is great!!

We need a python expert to come in here and explain to me what the various prefixes mean and such but at least other users can find this issue and see the pathway to working through it.