search

JuliaPy / pyjulia

python interface to julia
MIT License
886 stars 103 forks source link

PyJulia broken with Julia 1.5.2 Python 3.8? #425

Open opus111 opened 4 years ago

opus111 commented 4 years ago

Hello, I installed Julia 1.5.2 and tried to call it from Python. It failed on the "Julia( compiled_modules=False )"

To reproduce in Python call

>>> import julia
>>> julia.install()
>>> from julia.api import Julia
>>> j = Julia( compiled_modules=False )

Full log below

(ti) DRMB-P1K5HTD6:inquiries-proto peter.wolf$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> julia.install()
[ Info: Julia version info
Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  uname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64 i386
  CPU: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz: 
              speed         user         nice          sys         idle          irq
       #1  2900 MHz      50533 s          0 s      38657 s     114178 s          0 s
       #2  2900 MHz       8104 s          0 s       5792 s     189467 s          0 s
       #3  2900 MHz      43072 s          0 s      25683 s     134608 s          0 s
       #4  2900 MHz       7571 s          0 s       4409 s     191383 s          0 s
       #5  2900 MHz      40053 s          0 s      21347 s     141963 s          0 s
       #6  2900 MHz       7810 s          0 s       4184 s     191369 s          0 s
       #7  2900 MHz      37447 s          0 s      17718 s     148199 s          0 s
       #8  2900 MHz       8445 s          0 s       4099 s     190819 s          0 s

  Memory: 16.0 GB (786.87109375 MB free)
  Uptime: 20335.0 sec
  Load Avg:  2.4521484375  2.80029296875  3.0302734375
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_HOME = /Applications/Julia-1.5.app/Contents/Resources/julia
  TERM = xterm-256color
  PATH = /anaconda3/envs/ti/bin:/anaconda3/condabin:/Users/peter.wolf/.cargo/bin:/Applications/Julia-1.5.app/Contents/Resources/julia/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
  JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home
  XPC_FLAGS = 0x0
  HOME = /Users/peter.wolf
  JULIA_HOME = /Applications/Julia-1.5.app/Contents/Resources/julia
[ Info: Julia executable: /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia
[ Info: Trying to import PyCall...
┌ Warning: PyCall is already installed.  However, you may have trouble using
│ this Python executable because it is statically linked to libpython.
│ 
│ For more information, see:
│     https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
│ 
│ Python executable:
│     /anaconda3/envs/ti/bin/python
│ Julia executable:
│     /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia
└ @ Main /anaconda3/envs/ti/lib/python3.8/site-packages/julia/install.jl:90
>>> from julia.api import Julia
>>> j = Julia( compiled_modules=False )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/ti/lib/python3.8/site-packages/julia/core.py", line 503, in __init__
    self._call(IMPORT_PYCALL)
  File "/anaconda3/envs/ti/lib/python3.8/site-packages/julia/core.py", line 538, in _call
    self.check_exception(src)
  File "/anaconda3/envs/ti/lib/python3.8/site-packages/julia/core.py", line 587, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'LoadError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))
opus111 commented 4 years ago

For reference, here is the same code successfully running with Python 3.8 and Julia 1.4.2

>>> import julia
>>> julia.install()
[ Info: Julia version info
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  uname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64 i386
  CPU: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz: 
              speed         user         nice          sys         idle          irq
       #1  2900 MHz      53665 s          0 s      40822 s     119461 s          0 s
       #2  2900 MHz       9145 s          0 s       6361 s     198436 s          0 s
       #3  2900 MHz      45900 s          0 s      27262 s     140781 s          0 s
       #4  2900 MHz       8596 s          0 s       4887 s     200459 s          0 s
       #5  2900 MHz      42743 s          0 s      22734 s     148465 s          0 s
       #6  2900 MHz       8893 s          0 s       4658 s     200392 s          0 s
       #7  2900 MHz      40007 s          0 s      18925 s     155010 s          0 s
       #8  2900 MHz       9648 s          0 s       4583 s     199711 s          0 s

  Memory: 16.0 GB (56.6640625 MB free)
  Uptime: 21393.0 sec
  Load Avg:  4.06494140625  3.58154296875  3.1845703125
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_HOME = /Applications/Julia-1.4.app/Contents/Resources/julia
  TERM = xterm-256color
  PATH = /anaconda3/bin:/anaconda3/condabin:/Users/peter.wolf/.cargo/bin:/Applications/Julia-1.4.app/Contents/Resources/julia/bin:/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
  JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home
  XPC_FLAGS = 0x0
  HOME = /Users/peter.wolf
  JULIA_HOME = /Applications/Julia-1.4.app/Contents/Resources/julia
[ Info: Julia executable: /Applications/Julia-1.4.app/Contents/Resources/julia/bin/julia
[ Info: Trying to import PyCall...
┌ Warning: PyCall is already installed.  However, you may have trouble using
│ this Python executable because it is statically linked to libpython.
│ 
│ For more information, see:
│     https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
│ 
│ Python executable:
│     /anaconda3/bin/python
│ Julia executable:
│     /Applications/Julia-1.4.app/Contents/Resources/julia/bin/julia
└ @ Main /anaconda3/lib/python3.7/site-packages/julia/install.jl:74
>>> from julia.api import Julia
>>> j = Julia(compiled_modules=False)
>>>
tkf commented 4 years ago

This is likely the same as #424 and if so it would be solved by https://github.com/JuliaPy/PyCall.jl/pull/841

tkf commented 4 years ago

Ref https://github.com/JuliaRegistries/General/pull/22235

danvip10 commented 3 years ago

I'm getting the same error when not running julia.install() before. However, when I run julia.install() and then Julia(compiled_modules=False), I get a segmentation fault:

>>> from julia.api import Julia
>>> jl = Julia(compiled_modules=False)

signal (11): Segmentation fault: 11
in expression starting at none:0
PyUnicode_FromFormatV at /opt/anaconda3/bin/python (unknown line)
PyErr_Format at /opt/anaconda3/bin/python (unknown line)
_PyObject_FastCallKeywords at /opt/anaconda3/bin/python (unknown line)
call_function at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalFrameDefault at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalCodeWithName at /opt/anaconda3/bin/python (unknown line)
_PyFunction_FastCallKeywords at /opt/anaconda3/bin/python (unknown line)
call_function at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalFrameDefault at /opt/anaconda3/bin/python (unknown line)
function_code_fastcall at /opt/anaconda3/bin/python (unknown line)
call_function at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalFrameDefault at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalCodeWithName at /opt/anaconda3/bin/python (unknown line)
_PyFunction_FastCallDict at /opt/anaconda3/bin/python (unknown line)
slot_tp_init at /opt/anaconda3/bin/python (unknown line)
type_call at /opt/anaconda3/bin/python (unknown line)
_PyObject_FastCallKeywords at /opt/anaconda3/bin/python (unknown line)
call_function at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalFrameDefault at /opt/anaconda3/bin/python (unknown line)
_PyEval_EvalCodeWithName at /opt/anaconda3/bin/python (unknown line)
PyRun_InteractiveOneObjectEx at /opt/anaconda3/bin/python (unknown line)
PyRun_InteractiveLoopFlags at /opt/anaconda3/bin/python (unknown line)
PyRun_AnyFileExFlags at /opt/anaconda3/bin/python (unknown line)
pymain_main at /opt/anaconda3/bin/python (unknown line)
main at /opt/anaconda3/bin/python (unknown line)
Allocations: 19269207 (Pool: 19264329; Big: 4878); GC: 21
zsh: segmentation fault  python
danvip10 commented 3 years ago

Is there any fix for this?

danvip10 commented 3 years ago

I uninstalled and reinstalled both julia and PyCall, and although don't a segmentation error as before, I do get the same error message as @opus111 in the original post. Moreover, note that my python version is 3.7.6.

Could I fix this by updating any package through github?

vogt31337 commented 3 years ago

Maybe I can help a bit. But I think not... Using Julia 1.5.3 and python 3.8.6 together with conda (I know it's not supported, but with some symlinks it seems to work).

>>> julia.install()
[ Info: Julia version info
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  uname: Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64
  CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz:
                 speed         user         nice          sys         idle          irq
       #1-24  1200 MHz    4679934 s       1521 s    1741261 s  2493517096 s          0 s

  Memory: 31.1365966796875 GB (2360.359375 MB free)
  Uptime: 1.042066e6 sec
  Load Avg:  0.4296875  0.15673828125  0.091796875
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, haswell)
Environment:
  TERM = xterm
  ANT_HOME = /opt/rocks
  LD_LIBRARY_PATH = /opt/openmpi/lib
  PATH = /home/user/julia-1.5.3/bin:/home/user/miniconda3/envs/ppjl/bin:/home/user/miniconda3/condabin:/opt/openmpi/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/home/user/bin
  JAVA_HOME = /usr/java/latest
  MODULEPATH = /usr/share/Modules/modulefiles:/etc/modulefiles
  HOME = /home/user
  MPIHOME = /opt/openmpi
  PYTHONPATH = :/home/user/pandapower:
  MODULESHOME = /home/user/miniconda3/envs/pp/Modules/3.2.10
[ Info: Julia executable: /home/user/julia-1.5.3/bin/julia
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed and compatible with Python executable.
│
│ PyCall:
│     python: /home/user/miniconda3/envs/ppjl/bin/python
│     libpython: /home/user/miniconda3/envs/ppjl/lib/libpython3.8.so.1.0
│ Python:
│     python: /home/user/miniconda3/envs/ppjl/bin/python
└     libpython: /home/user/miniconda3/envs/ppjl/lib/libpython3.8.so.1.0
>>> from julia import Base

signal (11): Segmentation fault
in expression starting at none:0
_Py_DECREF at /usr/local/src/conda/python-3.8.6/Include/object.h:470 [inlined]
GetResult at /usr/local/src/conda/python-3.8.6/Modules/_ctypes/callproc.c:944 [inlined]
_ctypes_callproc at /usr/local/src/conda/python-3.8.6/Modules/_ctypes/callproc.c:1257
PyCFuncPtr_call at /usr/local/src/conda/python-3.8.6/Modules/_ctypes/_ctypes.c:4201
list_sort_impl at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/listobject.c:2453
compiler_addop_j at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/compile.c:1434
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_stmt at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:396 [inlined]
validate_stmts at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:523
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_stmt at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:396 [inlined]
validate_stmts at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:523
sre_ucs4_charset at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/sre_lib.h:169
_Py_ClearFileSystemEncoding at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/preconfig.c:35
list_sort_impl at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/listobject.c:2424
_codecs_utf_7_decode at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/clinic/_codecsmodule.c.h:406
mult at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/dtoa.c:653
convertsimple at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/getargs.c:1019 [inlined]
convertitem at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/getargs.c:601 [inlined]
vgetargs1_impl at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/getargs.c:391
validate_expr at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:249
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_stmt at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:396 [inlined]
validate_stmts at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:523
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_expr at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:328
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_expr at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:328
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
validate_expr at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:328
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1578
k_mul at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/longobject.c:3455
missing_arguments at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:3908
new_interpreter at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/pylifecycle.c:1528 [inlined]
Py_NewInterpreter at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/pylifecycle.c:1565
deque_new at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/_collectionsmodule.c:162
parse_internal_render_format_spec at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/formatter_unicode.c:182
_Py_ListComp at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/Python-ast.c:2072
sre_ucs4_charset at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/sre_lib.h:169
fixstate at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:106 [inlined]
fixdfa at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:58 [inlined]
PyGrammar_AddAccelerators at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:29
validate_expr at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ast.c:328
sre_ucs4_charset at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/sre_lib.h:169
fixstate at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:106 [inlined]
fixdfa at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:58 [inlined]
PyGrammar_AddAccelerators at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Parser/acceler.c:29
k_mul at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/longobject.c:3455
missing_arguments at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:3908
_Py_FinishPendingCalls at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:607
SubString_new_object_or_empty at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/stringlib/unicode_format.h:73
sre_ucs4_charset at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/sre_lib.h:169
new_arena at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1235 [inlined]
pymalloc_alloc at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Objects/obmalloc.c:1494
PyEval_EvalCode at python (unknown line)
unknown function (ip: 0x560427c96ca2)
unknown function (ip: 0x560427cb08f2)
import_all_from at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:5246 [inlined]
_PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:3017
import_all_from at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:5255 [inlined]
_PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:3017
_PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:1993
_Py_DECREF at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Include/object.h:478 [inlined]
_PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Python/ceval.c:2914
Py_BytesMain at python (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
PyInit_errno at /home/conda/feedstock_root/build_artifacts/python-split_1606502903469/work/Modules/errnomodule.c:210
Allocations: 3290498 (Pool: 3289132; Big: 1366); GC: 4
Segmentation fault

I tried this "test" multiple times: same result.

Then I tried your "code":

(ppjl) [user@pc ~]$ python
Python 3.8.6 | packaged by conda-forge | (default, Nov 27 2020, 19:31:52)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> from julia.api import Julia
>>> j = Julia(compiled_modules=False)
>>> from julia import Base
>>> Base.sind(90)
1.0

Et voilá it works. I don't know why, just started with Julia. But maybe this helps getting some insight in this problem. Btw. maybe now it's "fixed", and it runs, if you ignore compiled_modules. Maybe someone with more insight can provide help / details.

nclarkjudd commented 3 years ago

I am unable to replicate this issue.

Python 3.9.1 in a virtualenv, Julia 1.5.3 acquired through Fedora package manager, installed PyCall from Julia's REPL and then in IPython ran:

import julia julia.install() from julia import Base Base.sind(90)

These commands completed successfully.

Note I did not use conda here (at least, not in purpose --- I gather that PyCall uses conda under the hood somewhere, based on the output it generates).