Support virtual environments created by poetry/pipenv

[lccambiaghi]

Hi, first of all THANK YOU for your amazing work. I am a Data Scientist working with Python trying to start working with Clojure and your library is the most important reason that pushed me to attempt this jump (the second most important reason being Panthera, developed by Alan).

I am trying to set up your library with my python environment. I use pyenv to manage python versions and poetry to manage virtual environments.

So far my script looks like this:

(ns clj.core
(:require [libpython-clj.python :as py]))

(py/initialize! :python-executable "/Users/luca/git/experiments/.venv/bin/python"
                :library-path "/Users/luca/.pyenv/versions/3.7.6/libpython3.7m.a")

But it fails. The log says:

INFO: Executing python initialize with options:{:python-executable "/Users/luca/git/experiments/.venv/bin/python", :program-name nil, :python-home nil, :library-path "/Users/luca/.pyenv/versions/3.7.6/libpython3.7m.a"}
May 01, 2020 12:10:36 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Detecting startup-info for Python executable: /Users/luca/git/experiments/.venv/bin/python
May 01, 2020 12:10:36 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Startup info detected: {:python-home "/Users/luca/git/experiments/.venv", :lib-version "3.7", :libname "/Users/luca/.pyenv/versions/3.7.6/libpython3.7m.a", :java-library-path-addendum "/Users/luca/git/experiments/.venv/lib"}
May 01, 2020 12:10:36 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Setting java library path: /Users/luca/git/experiments/.venv/lib:/Users/luca/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
May 01, 2020 12:10:36 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Reference thread starting
Execution error (ExceptionInfo) at tech.jna.base/do-load-library (base.clj:158).
Failed to load library
May 01, 2020 12:19:15 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Executing python initialize with options:{:python-executable "/Users/luca/git/experiments/.venv/bin/python", :program-name nil, :python-home nil, :library-path "/Users/luca/.pyenv/versions/3.7.6/libpython3.7m.a"}
May 01, 2020 12:19:15 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Detecting startup-info for Python executable: /Users/luca/git/experiments/.venv/bin/python
May 01, 2020 12:19:15 PM clojure.tools.logging$eval12554$fn__12557 invoke
INFO: Startup info detected: {:python-home "/Users/luca/git/experiments/.venv", :lib-version "3.7", :libname "/Users/luca/.pyenv/versions/3.7.6/libpython3.7m.a", :java-library-path-addendum "/Users/luca/git/experiments/.venv/lib"}
Execution error (ExceptionInfo) at tech.jna.base/do-load-library (base.clj:158).
Failed to load library

How can I help you debug this problem?

[cnuernber]

I have a questions - What OS are you using?

That lib suffix- libpython3.7m.a - That looks like a static library. libpython-clj can only be used with shared libraries so those should have a .so or .dylib or .dll suffix. Is there a python library in there with that suffix?

I found this documentation on pyenv: https://github.com/pyenv/pyenv/wiki#how-to-build-cpython-with-framework-support-on-os-x

[lccambiaghi]

I compiled pyenv with support for shared libraries and updated the command to:

(py/initialize! :python-executable "/Users/luca/git/experiments/.venv/bin/python"
                :library-path "/Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib")

Now I get this error:

INFO: Library /Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib found at [:system "/Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"]
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

EDIT: oh I am on latest MacOS!

[cnuernber]

OK, so that encodings library may require another shared library.

What I had to for Conda was set LD_LIBRARY_PATH before I start clojure:

https://github.com/clj-python/libpython-clj/blob/master/scripts/conda-repl

[jjtolton]

Additionally you might try not including :library-path. Your :python-executable will inform libpython-clj where its library path is, so try removing that parameter.

[lccambiaghi]

@cnuernber I have tried to export that env variable but I still get:

...
INFO: Library /Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib found at [:system "/Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"]
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
...

@jjtolton that also did not work:

clojure.lang.ExceptionInfo: Failed to load library
{:libname "python3.6m", :paths [[:system ["python3.6m"]] [:java-library-path ["/Users/luca/git/experiments/.venv/lib/libpython3.6m.dylib" "/Users/luca/Library/Java/Extensions/libpython3.6m.dylib" "/Library/Java/Extensions/libpython3.6m.dylib" "/Network/Library/Java/Extensions/libpython3.6m.dylib" "/System/Library/Java/Extensions/libpython3.6m.dylib" "/usr/lib/java/libpython3.6m.dylib" "./libpython3.6m.dylib"]]]}
 at tech.jna.base$do_load_library.invokeStatic (base.clj:158)
    tech.jna.base$do_load_library.invoke (base.clj:123)
    tech.jna.base$load_library.invokeStatic (base.clj:178)
...

[jjtolton]

@lccambiaghi I have some time this weekend, I'll try to dive in a little deeper.

[cnuernber]

@lccambiaghi - in addition the one pathway we have that is repeatable and works is to use a dockerized development setup.

Have you seen the facial rec demo? In that case I use a docker container to ensure that issues like this one don't come up. Really that is a best pathway; use docker. I don't believe that anything else is likely to work reliably especially when you take your code into production or attempt to make it work across multiple machines. In addition, if it doesn't work, you have a repeatable case we can all look at. Please consider it :-); the number of python environments is infinite; we have to reduce that compatibility matrix.

[lccambiaghi]

@jjtolton thank you so much for dedicating time to this, I appreciate it a lot. Please let me know if I can help you in any way. For example, I could set up a docker image with pyenv+poetry+clojure to reproduce the error if you want me to.

@cnuernber you are absolutely right, I would never go to production without a docker image and a reproducible build, I'm not quite there yet :) I have set up a conda environment and it works just out of the box. This issue is just a "would be really nice if...", I was hoping that I was doing something stupid, for example your suggestion of compiling with shared library was spot on!

The conda support is working already very well. Some people in the python software engineering community like a lighter tool for virtual environments and poetry is gaining a lot of traction, replacing pipenv. Poetry and conda don't behave very well on the same system if you are not careful about your PATH. Supporting both would attract (even) more python developers to try out your amazing library! But maybe it is just me who thinks using a docker image is too much work.. I should maybe just find time to get confidence with docker+TRAMP+cider :)

[cnuernber]

That sounds good, the thing is in your instance I have no idea why it isn't working further and a lot of times these issues boil down into something specific to that user's machine. I agree it would be nice if everything worked.

You don't need TRAMP, though. In our example I just use straight docker and cider by mounting the local directory. This makes all the files available to your repl. In addition I run the container as the current user so that means that it reads/writes files as you (at least on ubuntu and I believe MacOS). TRAMP makes things a bit harder I believe.

[lccambiaghi]

Ah that sounds easier than I expected, thanks! The only (very minor) problem I have is that when I connect to a nREPL server (instead of doing cider-jack-in) I don't see the function docs in the minibuffer. Do you by chance have a solution for that as well?

[cnuernber]

I believe that is fixed by the way lein is launched in the docker container:

My solution was to cut/paste the command that appears in the emacs message buffer when you do 'cider-jack-in' :-).

https://github.com/cnuernber/facial-rec/blob/master/scripts/conda-repl#L9

[jjtolton]

@lccambiaghi just curious from your comments are you more interested in poetry OR pyenv? just for the sake of not trying to tune both if you only need one.

[lccambiaghi]

@cnuernber ha, that is smart! Would you mind sharing your cider-connect-clj function? Mine currently looks like this:

(defun cider-connect-clj (&optional params)
  "Initialize a Clojure connection to an nREPL server.
PARAMS is a plist optionally containing :host, :port and :project-dir.  On
prefix argument, prompt for all the parameters."
  (interactive "P")
  (cider-nrepl-connect
   (thread-first params
     (cider--update-project-dir)
     (cider--update-host-port)
     (cider--check-existing-session)
     (plist-put :repl-init-function nil)
     (plist-put :session-name nil)
     (plist-put :repl-type 'clj))))

I will set up a Dockerfile with clojure+pyenv+poetry.

@jjtolton I don't have a strong preference :) one complements the other.

[cnuernber]

That is interesting. I just do M-x cider-connect - localhost - port#.

If you could build out a docker file under libpython-clj/dockerfiles and build/run scripts under libpython-clj/scripts that would allow us to show new people how to do this.

[jjtolton]

Haven't had a chance to check yet but it's possible that release 1.44 will fix the issues with pyenv/poetry. When I was poking around it looked like the libpython for pyenv did not conform to the regex we had at the time. Perhaps @lccambiaghi you can give 1.44 a shot? If not I'll poke around this evening.

[lccambiaghi]

@cnuernber The cider-connect-clj definition I pasted is the standard one defined in cider and it is the one binded by Doom Emacs. I tried to just use the standard cider-connect as suggested by you but I still have the same problem that eldoc does not suggest me function docs in the echo area and I cannot call cider-doc on the symbol at point.

I will open a PR for the Dockerfile, happy to contribute after your kind help!!

@jjtolton still getting No module named 'encodings'. Here the startup info detected:

Startup info detected:
{:lib-version "3.7",
 :java-library-path-addendum "/Users/luca/git/docker-py-clj/.venv/lib",
 :exec-prefix "/Users/luca/git/docker-py-clj/.venv",
 :executable "/Users/luca/git/docker-py-clj/.venv/bin/python",
 :libnames
 ("/Users/luca/.pyenv/versions/3.7.6/Python.framework/Versions/3.7/lib/libpython3.7m.dylib"
  "python3.7"),
 :prefix "/Users/luca/git/docker-py-clj/.venv",
 :base-prefix "/Users/luca/.pyenv/versions/3.7.6",
 :base-exec-prefix "/Users/luca/.pyenv/versions/3.7.6",
 :python-home "/Users/luca/git/docker-py-clj/.venv",
 :version [3 7 6],
 :platform "darwin"}

[lccambiaghi]

I have set up a quick repo here: https://github.com/lccambiaghi/docker-py-clj , I will create the scripts and open the PR soon. Should I use lein instead of tools.deps?

Well, the news is that pyenv+poetry in Docker WORKS! So it is something strange in my setup, you were right! Maybe something MacOS specific? The only obvious difference is the pyenv .dylib file, which in Linux is a .so file.

[lccambiaghi]

Ok I managed to fix the issue! I needed to set one environment variable called PYTHONHOME.

If this is my command:

(py/initialize! :python-executable "/Users/luca/git/docker-py-clj/.venv/bin/python" :library-path "/Users/luca/.pyenv/versions/3.7.6/lib/libpython3.7m.dylib")

I had to set it to:

export PYTHONHOME="/Users/luca/.pyenv/versions/3.7.6"

And then libpython-clj was able to import encodings.

EDIT: actually, with the env variable set up, I don't really need to call the py/initalize! command but it detects correctly the python environment! Great job

[jjtolton]

This is great news! pyenv/poetry has been on my list of annoying issues for a little bit so I'm glad it got worked out :) Thank you!

[cnuernber]

Yes, that is encouraging for sure. We try to detect 'pythonhome' when we do the system info lookup, it gets encoded in python as prefix I think and I thought would get auto-inferred from the running python executable but I guess that pathway failed on your mac for some reason.

[lccambiaghi]

Happy to help, hopefully it can help some future user to solve the same issue! Let's just be sure to document this corner case and not bury it in the closed issues :)