Draft of pyenv+poetry Dockerfile + scripts

[lccambiaghi]

Opening this WIP pull request to gather some feedback! I saw there is some ongoing work to support py38. In principle pyenv can be used to support any python version through docker args.

The main advantage of poetry over conda are the pyproject.toml + poetry.lock files, the better dependency resolution and reproducible builds. But maybe this is only a wanted feature for complex python projects?

I also rephrased the build and run scripts together with the docker file, not sure if you want me to pollute the scripts folder?

[lccambiaghi]

Ahah, sorry to waste your time like this Chris! It was a WIP commit I pushed last night before returning back to it this morning! The problem is that when we mount the pwd in the docker container the deps.edn in your repo superseded the one echoed in the Dockerfile. I moved the alias to the repo if you don't mind it.

I am taking inspiration by this Dockerfile: https://github.com/scicloj/docker-hub/blob/master/libpython-clj/Dockerfile

I still did not manage to connect CIDER to the docker nREPL server. Furthermore, if I start a standard REPL inside the container, I cannot require libpython-clj:

user=> (require '[libpython-clj.require :refer [require-python]])
Syntax error (ClassNotFoundException) compiling at (libpython_clj/jna/base.clj:1:1).
libpython_clj.jna.PyObject

I also need to edit my Dockerfile not to write anything in /home/app/ but I should instead do the development in /home/libpython-clj.

[cnuernber]

No worries, good point about the aliasing of deps.edn I didn't think of that. LMK when is a good time to take it for a spin.

[lccambiaghi]

I think the Dockerfile is potentially ready, except for compacting the commands and env variables to have fewer layers.

I still have the two problems mentioned above: if I simply execute bash scripts/pyenv-poetry-repl I cannot use libpython-clj. Maybe it is not fully installed with deps.edn?

The second is with some docker connectivity. If I run bash scripts/pyenv-poetry-repl I can connect CIDER with it, however if I start the repl with the docker image I cannot connect to it. I also tried to expose 8888 but I cannot seem to connect. Maybe we can ask Alan for help troubleshooting since I am mocking hist Dockerfile?

[cnuernber]

One problem is that libpython-clj has pretty significant .java files and deps.edn does ot work with projects with java. A few aspects of jna (the c-interop layer) are easier in java or require a java class with a specific layout or specific attributes on functions.

This is one reason why libpython-clj has a hybrid deps/project structure. So for the poetry case, you will need include libpython-clj by maven coordinates and not use the local deps.edn file.

If you try to use libpython-clj from a pure deps edn approach you get:

user> (require '[libpython-clj.python :as py])
Syntax error (ClassNotFoundException) compiling at (libpython_clj/jna/base.clj:1:1).
libpython_clj.jna.PyObject
user> 

In a downstream project like Panthera that isn't a problem like of course and I really like the deps pathway assuming your project doesn't need to compile java into class files.

As for connecting that is a good question and the networking side of docker/nrepl/cider is IMO pretty tough. I see you specifically set the host to 0.0.0.0 which should allow connections from any location but on these things I get stuck frequently also. In addition the docker run command has --net=host which means docker should use the host's networking system which should also help...

Perhaps we should move all the poetry stuff to something like env/poetry where the main library setup and deps.edn won't interfere with the poetry setup. We can also move the conda and pything3.8 setup to similar directories and just a use a flat directory structure with the scripts and everything in one place. I believe that will mitigate a lot of the issues popping up here as long as the deps pathway does in fact include the python library by maven coordinates and not by git sha.

If you want to try that specifically with poetry then once that is working I can move the other two pathways. This may also be easier to follow if you really just want to learn/copypasta a poetry setup.

[cnuernber]

Closing this :-). I think there is room to have more user-focused dockerfiles but for now I think this isn't necessary.