Open tkf opened 6 years ago
This was very helpful; thank you, @tkf
I must say I was very-very surprised by not finding libpython3
in my ldd $(which python3)
output.
Debian/Ubuntu packagers are the known strong advocates against linking things statically; but doing so in python/3
?!?!
Yeah, I was surprised too. I thought it was just conda (in which static compilation maybe makes sense).
@tkf Thanks for your reply. But there are still some questions:
Thanks.
I think there are three ways to use Python with Julia 1.0 on Ubuntu/Debian/etc. at the moment:
IPython.jl/src/core.jl
.Will this issue be solved in near future
I think I know how to fix it #173. But likely not in next weeks. I can't grant anything :) If I were to do it I want to do it after dropping Julia 0.6 (which requires PyJulia to be released soon).
I made an experimental package julia-venv
: https://github.com/tkf/julia-venv
It implements the private DEPOT_PATH
hack mentioned in #173 but does so per-virtualenv basis. This let us use different Python executables without recompiling PyCall.jl all the time. On top of creating a private DEPOT_PATH
, it installs a CLI julia-venv
which acts like the normal julia
interpreter (by invoking it via C API from Python). This way, we can run PyCall.jl always in "PyJulia-mode" so that the issue with Python in Ubuntu/Debian and Conda can be avoided (https://github.com/tkf/julia-venv#how-it-works). As a nice side-effect, you can also use PyCall.jl with different Python executable and different set of Python libraries.
Edit: Actually, Edit^2: It works now.julia-venv
probably still doesn't work with Conda (and probably also with Ubuntu): https://github.com/tkf/julia-venv/pull/2
@tkf ,thanks for your awesome work. The new package works on Ubuntu 16.04 with anaconda. It really helps.
There are some minor suggestions (or some misunderstanding of me).
for test purpose, I test from julia import NPZ
, while NPZ
is already added in original julia environment. But it failed.
and the following code works:
from julia import Pkg
Pkg.add("NPZ")
#... some install outputs
from julia import NPZ
data = NPZ.npzread("path")
It seems that pyjulia and julia of system is using different package cache? I think re-install packages is not a problem and it might be a valuable feature. I only suggest add some doc for this
broadcast is commonly used in julia, but Main.abs.([1,2,3])
is a SyntaxError
in python.
in README.md
of pyjulia, there is
Main.eval("sin.(xs)")
which seems that pyjulia will not expose broadcast method to python level. I'm not sure if I am right, but if it is true, I'm thinking is it possible to adding some special syntax like Main.abs.broadcast([1,2,3])
or in short Main.abs.b([1,2,3])
(although may introduce some confusing to sum_b
for python version sum!
).
InterruptException()
handlingWhen I press ctrl-C
in python REPL, there would be a small problem.
If I'm using a pure python REPL, it should work like this
>>> # presss ctrl-C here
KeyboardInterrupt
>>>
But in a python REPL which loaded julia, it would work like this:
>>> # press ctrl-C here
and nothing happens, after that if I enter next command
>>> # Ctrl - C, and nothing happens
>>> Main.xs = [1,2,3] # Enter, change line and type new command and Enter I got:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 140, in __setattr__
self._julia.eval(setter)(value)
File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 606, in eval
ans = self._call(src)
File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
self.check_exception(src)
File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
.format(exception, src))
julia.core.JuliaError: Exception 'InterruptException' occurred while calling julia code:
PyCall.pyfunctionret(
(x) -> eval(Main, :(xs = $x)),
Any,
PyCall.PyAny)
It seems the InterruptException
is send to julia in one delayed statement?
@Hong-Xiang Hey, thanks for the feedback. Are you using julia-venv
in Ubuntu or Debian? Did it actually work?
I think re-install packages is not a problem and it might be a valuable feature.
Yeah, something like virtualenv
's --system-site-packages
would be nice. I actually did that in first versions but stopped doing that since I encountered with some quirks in julia
's package handling. It it actually as "easy" as replacing depots = filter(!isequal(default_depot1), __py_julia_venv_orig_DEPOT_PATH)
with depots = copy(__py_julia_venv_orig_DEPOT_PATH)
.
convenient broadcast call
I started experimenting more numpy-like interface on top of PyJulia (i.e., broadcast operations on arrays by default):
https://ipyjulia-hacks.readthedocs.io/en/latest/
Not sure if it is suitable for PyJulia, though.
But if you want to leverage Julia, I think it's better to just use Julia syntax. If you don't want to use global variable (of course), you can do:
Main.eval("(xs) -> sin.(exp.(xs) .+ 1)")(xs)
or equivalently
Main.eval("(xs) -> @. sin(exp(xs) + 1)")(xs)
Unlike numpy.exp
etc., it doesn't allocate intermediate arrays.
InterruptException
Hmm... I didn't notice that. But yes, I can reproduce it. Actually, I just noticed that it's reported in #189. Let's track the issue there.
It looks like IPython 7 (development version) handle ctrl-C differently and it doesn't have this problem. I've been using it so I didn't notice the problem.
@tkf my system/environment info is:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
and actually it's Ubuntu Mate.
conda:
$ conda --version
conda 4.3.30
python(in conda virutal env, with julia-venv
)
$ source activate juliaenv
(juliaenv) $ which python
(home path)/anaconda3/envs/juliaenv/bin/python
(juliaenv) $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia_venv import get_julia
>>> get_julia()
<julia.core.Julia object at 0x7f66c6cc7f28>
>>> from julia import Main
it worked, without error.
python(in conda virtual env, without julia-venv
)
(juliaenv) $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Main
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 172, in load_module
JuliaMainModule(self, fullname))
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 77, in __init__
self._julia = loader.julia
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 164, in julia
self.__class__.julia = julia = Julia()
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 496, in __init__
"\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
self.check_exception(src)
File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
.format(exception, src))
julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:
unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot(),"lib", "pyjulia3-v$(VERSION.major).$(VERSION.minor)"))
>>>
tests like using standard libraries, basic array operations and even third-party packages like NPZ
is performed, and it seems just worked, as mentioned above.
Is there any further tests I can do for this?
Thanks. Interesting... Actually, the problem I thought I had was already fixed https://github.com/tkf/julia-venv/pull/2. My guess was that it was some kind of IO caching issue (compilation file was not properly written before exiting the problem). It then maybe explains why it's machine-dependent.
Anyway, I'm glad that it worked in your machine!
For Ubuntu/Debian users, probably the cleanest and relatively easy way to use PyJulia at the moment is to install (build) Python using pyenv. But you need to pass --enable-shared
manually.
$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6
Downloading Python-3.6.6.tar.xz...
-> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
Installing Python-3.6.6...
Installed Python-3.6.6 to /home/takafumi/.pyenv/versions/3.6.6
200.54s user 16.69s system 271% cpu 1:20.10 total
$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6
linux-vdso.so.1 => (0x00007fff6933c000)
libpython3.6m.so.1.0 => /home/takafumi/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca44a6e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca446a4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca444a0000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fca4429d000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca43f94000)
/lib64/ld-linux-x86-64.so.2 (0x00007fca451c0000)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
Or you can switch to Arch Linux :laughing:
It uses --enable-shared
: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python&id=68e188a1f975dcc6798072bcda4ef35abfe162ed#n57
@ExpandingMan @mprogram So it turned out static linking is just the default behavior of Python's ./configure
script (see my message above https://github.com/JuliaPy/pyjulia/issues/185#issuecomment-420521153). I guess Debian is not doing anything particularly evil here.
Thanks, @tkf for your last two comments. I'm a fan of deb-based system and my thought would be that Debian needs to look at those. If I'll be up to recompile Python (for the sake of optimization with MKL, for example as Julia does), I would include the --enable-shared
option in my build.
@mprogram It looks like pyenv does something more than passing --enable-shared
to ./configure
:
I didn't check if ./configure --enable-shared
was enough when building without pyenv
.
This problem is not going to go away, Debian is not going to change it and lots and lots of people are going to try this as debian distros are the most popular (though I prefer Arch I still have to use them often myself). My first question is how other languages solve this problem. We can't be the first people ever to want to use a dynamically linked Python executable. It would be good to see how Scala and Java solve this, though I haven't started looking yet.
I think at a minimum what we need is a very clear set of the simplest possible steps that one would take to get pyjulia
running on debian in a very visible location right in the packages readme or documentation. Thanks to @tkf to taking the first steps to this with his pyenv setup, I'm going to try it myself when I get a chance.
Yeah "switch to Arch Linux" was rather a joke... :wink: It definitely is the biggest TODO in this package to support Debian family out-of-the-box.
My first question is how other languages solve this problem.
I don't think the question is how to solve the problem. In principle there is a straightforward solution: just compile different cache of PyCall.jl and all other packages depending on it to different files. For example, here is an idea to implement it: https://github.com/JuliaLang/julia/issues/27418#issuecomment-417826061. The question is when that happens. Or maybe it's more like if we can convince Julia core devs to include enough features required by PyJulia.
Until then, here is yet another solution: #200. The idea is to bundle python-jl
executable in PyJulia package which can be used to run your Python script instead of the usual python
executable.
I think I'm rather confused about what the issue is here. I thought the problem was that since the debian Python binary is statically linked, using libpython
doesn't do you any good because you can't interact with the process started by python
. What I don't understand is what this has to do with the pre-compile process on the Julia side. I would think that pyjulia
is just talking to the julia
process with libjulia
and that you'd be perfectly able to use everything that was already precompiled for your regular Julia process. Care to clarify? (reading this thread did not clear it up for me)
I thought the problem was that since the debian Python binary is statically linked, using libpython doesn't do you any good because you can't interact with the process started by python.
You can interact with C API of statically linked python
. For example, Python itself provides ctypes.pythonapi
for such purpose. Actually, python
statically linked to libpython
worked in julia
0.6 because it was easier to monkey-patch precompilation mechanism of julia
0.6.
The reason why it's related to Julia's precompilation mechanism is explained in julia/fake-julia/README
. If you look at this if
block in src/startup.jl
in PyCall.jl
, you can see that it generates different macros depending on libpython is statically linked or not (libpython === nothing
or not). This if
block is run at compile time and those macros are used all over the places so the functions in PyCall.jl
becomes incompatible for the program statically linked to libpython (e.g., python
in Ubuntu) and the program dynamically linked libpython (julia
). Apparently defining those different set of macros was important for improving the load time of PyCall.jl
(https://github.com/JuliaPy/PyCall.jl/issues/167) so I guess we can't do the branching at runtime in PyCall.jl
(I'm mentioning it since that was my question: https://github.com/JuliaPy/PyCall.jl/issues/528).
I merged a workaround for statically linked Python #200. Basically, you "just" need to run python-jl script.py
(notice the -jl
suffix) instead of python script.py
if you use Debian or conda. You can try it by:
pip install https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia
Awesome job! Thanks so much for your work on this. This was a really awkward problem in which we were basically just getting screwed by the distros (and sadly some of the most important ones) and there was really never going to be a perfect solution, this seems to be about the best we'll be able to do with the situation we've been put in.
If you could, I think it would be really great if you could explain this as much as possible in the README (I won't offer to do it myself since I'm not sure I fully understand what you did yet).
Thanks again!
Yep, README is the only blocker before the release ATM.
Guys I might have found a solution to this! If you install python from conda-forge there is no need for python-jl
. Try this
conda install -c conda-forge python
python -c "import julia.Main"
It should work. See https://github.com/JuliaPy/pyjulia/pull/212#issuecomment-433229160
How is that a better solution than python-jl
? Personally I'd prefer python-jl
running on the system Python binaries, but I suppose it's nice to have as many options as possible.
If you are using Conda you are already not using the system Python binaries. In the other cases, I agree there is still need for python-jl
. Also, there can still be issues with python-jl
https://github.com/JuliaPy/pyjulia/issues/208.
Agreed, let's just aim to document as many options as possible and people will be able to select whatever solution works best for them.
@bstellato Thanks!. This is a good news! So I guess conda-forge
distributes a Python executable dynamically linked to libpython. I'll add it to the documentation.
@ExpandingMan Yeah, I think python-jl
should stay. Another benefit of python-jl
is that signal handler is slightly well behaved this way (libjulia "steals" signal handler from python
#211).
It's a bit strange. 3.7.0 from conda-forge is statically linked:
$ conda create --prefix conda-forge-python conda-forge::python
...
The following NEW packages will be INSTALLED:
...
python: 3.7.0-h5001a0f_4 conda-forge
...
$ conda-forge-python/bin/python
Python 3.7.0 | packaged by conda-forge | (default, Sep 30 2018, 14:56:18)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ ldd conda-forge-python/bin/python
linux-vdso.so.1 (0x00007ffec5dce000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f86eb290000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f86eb28b000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f86eb286000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f86eb27c000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f86eb0f7000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f86eaf33000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f86eb2e4000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
None
But conda-forge::python=3.6
is dynamically linked.
$ rm -r conda-forge-python
$ conda create --prefix conda-forge-python conda-forge::python=3.6
...
The following NEW packages will be INSTALLED:
...
python: 3.6.6-h5001a0f_3 conda-forge
...
$ ldd conda-forge-python/bin/python
linux-vdso.so.1 (0x00007ffff0f7e000)
libpython3.6m.so.1.0 => /home/takafumi/repos/pyjulia/conda-forge-python/bin/../lib/libpython3.6m.so.1.0 (0x00007f1f758a8000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1f75856000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1f75851000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f1f7584c000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f1f75842000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f1f756bd000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f1f754f7000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1f75de7000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/home/takafumi/repos/pyjulia/conda-forge-python/lib/libpython3.6m.so.1.0
Version 3.7.0 is still dynamically linked for me on mac os.
conda create -n testpyjulia37 conda-forge::python
conda activate testpyjulia37
otool -L $(which python)
I get
/Users/xxx/miniconda3/envs/testpyjulia37/bin/python:
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
@rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
Also,
> python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/Users/xxxx/miniconda3/envs/testpyjulia37/lib/libpython3.7m.dylib
Interesting. I asked the question in conda-forge: https://github.com/conda-forge/python-feedstock/issues/222
I wrote README in #210 and you can read it here: https://github.com/tkf/pyjulia/blob/readme/README.md
@ExpandingMan If you are interested, have a look at Troubleshooting and pre-compilation mechanism sections: https://github.com/tkf/pyjulia/blob/readme/README.md#troubleshooting https://github.com/tkf/pyjulia/blob/readme/README.md#pre-compilation-mechanism-in-julia-10
I just merged #256 to master. I think PyJulia is now usable with many CPython executables if you can compile a custom system image with python3 -m julia.sysimage
command. The bonus point is that this makes PyJulia startup virtually instantaneous. See: https://pyjulia.readthedocs.io/en/latest/sysimage.html
Hi @tkf - it's annoying having to deal with this issue. Do you know which Linux distro (ideally docker container) that provides a dynamically linked Python?
It works in Arch Linux :)
IIRC, the official python
docker works. It's a bit old, but here is an example for setting up PyJulia in Docker: https://github.com/tkf/docker-pyjulia
Also, just FYI, I started working on compile-time preference integration for PyCall https://github.com/JuliaPy/PyCall.jl/pull/835. If it works as I expected, PyJulia should just work without any hacks in Julia 1.6 and above. Obviously, there are still some unknowns as I haven't finished implementing all. So, let's see ;)
What's the status of this issue? I am trying to run PyJulia in Ubuntu 22 but python is statically linked.
Is it possible to build my own Python with --enable-shared
in Conda?
I think that might be a question for conda-forge. If you are going to compile it yourself, do you actually need conda though?
Perhaps we could use https://github.com/JuliaBinaryWrappers/Python_jll.jl
I'm writing down the issue so that it is easy to find by the users.
At the moment, for PyJulia to load Julia >= 0.7, two conditions have to be met:
Python interpreter configured for PyCall.jl and Python interpreter used to import PyJulia have to use identical
libpython
(once #190 is merged)have exactly identical paths.The Python interpreter have to be dynamically linked (which is usually the case in Windows and macOS according to this comment)
To check if condition 2 is the case, run:
where
/usr/bin/python
is the appropriate Python interpreter.If it does not print anything, you can't use PyJulia with this Python interpreter.