Allow any Python choice at runtime like PythonCall?

[marius311]

I'm not an expert and don't know the internals, but is there a reason PyCall can't do whatever PythonCall does that lets the user choose any Python executable at runtime? Is there anything preventing whatever they're doing there to be used here?

[stevengj]

Sure, in fact we originally did that too. However, we changed to determining it at build time to improve load times (#169).

Currently, PyCall and PyPlot load significantly faster than PythonCall and PythonPlot. I’m not sure if it’s primarily because of the libpython linkage or some other issues, however.

[marius311]

Thanks, didn't now that. Don't know if there's the bandwidth for it but it would be great to have this back as an option. Kind of like PYCALL_JL_RUNTIME_PYTHON but which let you do any Python whatsoever.

[jlapeyre]

@marus311 I wrote this https://github.com/jlapeyre/CachePath.jl and made a small change to PyCall in this branch https://github.com/jlapeyre/PyCall.jl/tree/deps-in-dir . This allows PyCall to work with any python without recompilation.

I'm pretty sure that it is not enough to allow you to use a python that is statically linked to libpython. There may be a way to get that too.

Here is a discourse post about it.

EDIT: One of the examples in the my discourse post seems like it is using a statically linked python, because its a conda installation. But, I'm not sure about that.

[jlapeyre]

I just checked again, and it looks like PyCall is working with a statically linked libpython:

julia> const PyCall = CachePath.require("PyCall", "../adepot")
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
PyCall

julia> PyCall.@pyimport sys

julia> sys.executable
"/home/lapeyre/.julia/conda/3/bin/python"

shell> ldd /home/lapeyre/.julia/conda/3/bin/python
    linux-vdso.so.1 (0x00007ffe5f1ba000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4110b0a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f411093b000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f4110934000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f411092f000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f4110924000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f41107e0000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4110eb1000)

I can't recall which package did that conda installation. Maybe it was PythonCall. In any case it's statically linked.

[sairus7]

By the way, static linking with absolute paths to python is the reason why we can't have portable julia apps with PyCall, see here: https://github.com/JuliaPy/PyCall.jl/issues/981

The path is just burned in binaries.

The only workaround I found so far is to set user-indefinite absolute paths (and copy conda into that path) before compilation, like this:

julia -e 'import Pkg; 
ENV["CONDA_JL_HOME"] = "/opt/conda/3";
ENV["PYTHON"]="/opt/conda/3/python3.9"; 
Pkg.add("PyCall"); 
Pkg.build("PyCall")'

And then take the whole coda folder into my julia app distro and copy it into the same path on installation. That leads to +2.6 GB (!!) and thousands of files...