custom system image

[andrewning]

I've been trying to create a custom system image on OS X with conda python. It seems to build fine. However, pyjulia gives me an error:

RuntimeError: It seems your Julia and PyJulia setup are not supported.

Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    /Users/aning/miniconda3/bin/python
    /Users/aning/miniconda3/lib/libpython3.7m.dylib
Python interpreter used to import PyJulia and its libpython.
    /Users/aning/miniconda3/bin/python
    /Users/aning/miniconda3/lib/libpython3.7m.dylib

I can use the compiled_modules=False just fine, but it is of course quite slow. I'm only Julia 1.1.

[tkf]

Just quickly reviewing the code, it could actually be a bug in PyJulia. For the moment, you may be able to workaround this issue by invoking low-level julia.api.LibJulia API: https://pyjulia.readthedocs.io/en/latest/api.html#julia.api.LibJulia Something like

from julia.api import LibJulia
api = LibJulia.load()
api.sysimage = "PATH/TO/CUSTOM/sys.so"
api.init_julia()

from julia import Main

[andrewning]

that worked!

[NLaws]

I am using a custom system image using the helpful tip provided above:

from julia.api import LibJulia
api = LibJulia.load()
api.sysimage = "PATH/TO/CUSTOM/sys.so"
api.init_julia()

from julia import Main

Which works on my Macbook with OS 10.14.6. However, on Red Hat Enterprise Linux Server release 7.8 (Maipo), I get the following traceback:

"<class 'julia.core.JuliaError'>", 'JuliaError("Exception \'ArgumentError\' occurred while calling julia code:\\n
using PyCall",)', 
['  File "/run_jump_model.py", line 105, in run_jump_model\n    from julia import Main\n', '  
File "<frozen importlib._bootstrap>", line 971, in _find_and_load\n', '  
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked\n', '  
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked\n', '  
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 246, in load_module\n    JuliaMainModule(self, fullname))\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 148, in __init__\n    self._julia = loader.julia\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 238, in julia\n    self.__class__.julia = julia = Julia()\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 502, in __init__\n    self._call(u"using PyCall")\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 536, in _call\n    self.check_exception(src)\n', '  
File "/env/lib/python3.6/site-packages/julia/core.py", line 586, in check_exception\n    .format(exception, src))\n'])

Some Googling indicates that this has been an issue with older versions of Julia (I'm using 1.4.2), but was fixed with either Julia updates or PyJulia updates. @tkf Any ideas on this issue? (The operating system issue seems strange so I could be doing something wrong on the Linux server, but I believe that I have everything set up the same in terms of python and julia on the macbook and the linux server).

[tkf]

Hmm.... It's hard to tell from that stacktrace. Maybe try something like

api.jl_eval_string(b"""try using PyCall; catch err; @error "using PyCall" exception = (err, catch_backtrace()); end""")

before from julia import Main?

[NLaws]

I put in the jl_eval_string as you suggested @tkf however that doesn't error. It gets to the next line from julia import Main and I get the same traceback :( I'll keep trying to get more useful information ...

[tkf]

That's strange... You can also try from julia.api import Julia and Julia(debug=True) before from julia import Main though I'm not sure how useful it'd be.

[NLaws]

I just ran the last suggestion via python CLI while ssh'ed into the server and I get the same error:

>>> import os
>>> from julia.api import LibJulia
>>> julia_img_file = os.path.join("julia_envs", "Xpress", "JuliaXpressSysimage.so")
>>> api = LibJulia.load()
>>> api.sysimage = julia_img_file
>>> api.init_julia()
[ Info: Xpress: Found license file /home/nlaws/xpressmp/bin/xpauth.xpr
[ Info: Xpress: Development license detected.
>>> from julia.api import Julia
>>> Julia(debug=True)
DEBUG 
DEBUG Debug-level logging is enabled for PyJulia.
DEBUG PyJulia version: 0.5.3
DEBUG 
DEBUG exception occured? 140415745573376
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/env/lib/python3.6/site-packages/julia/core.py", line 502, in __init__
    self._call(u"using PyCall")
  File "/env/lib/python3.6/site-packages/julia/core.py", line 536, in _call
    self.check_exception(src)
  File "/env/lib/python3.6/site-packages/julia/core.py", line 586, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'ArgumentError' occurred while calling julia code:
using PyCall

Maybe something to do with PyCall not in the Julia system image? Here is how I'm building the image:

using Pkg
using PackageCompiler

function build_julia_image(project_path::String)

    Pkg.activate(joinpath(project_path, "julia_envs", "Xpress"))
    Pkg.instantiate(verbose=true)

    # must point pycall to the python path we want it to use
    ENV["PYTHON"] = joinpath(project_path, "env", "bin", "python")
    println("Python path for PyCall: ", ENV["PYTHON"])
    Pkg.build("PyCall")
    Pkg.build("Xpress")
    include(joinpath(project_path, "julia_envs", "Xpress", "precompile.jl"))

    if Sys.islinux()
        ext = ".so"
    elseif Sys.isapple()
        ext = ".dylib"
    elseif Sys.iswindows()
        ext = ".dll"
    else
        error("Unsupported operating system")
    end

    PackageCompiler.create_sysimage(
        [:AxisArrays, :JuMP, :MathOptInterface, :PyCall, :Xpress, :MutableArithmetics],
        sysimage_path=joinpath(project_path, "julia_envs", "Xpress", "JuliaXpressSysimage" * ext),
        precompile_execution_file=joinpath(project_path, "julia_envs", "Xpress", "precompile.jl")
    )
end

where precompile.jl contains:

using JuMP
using AxisArrays
using MathOptInterface
using LinearAlgebra
using MutableArithmetics
using Printf
using PyCall

include(joinpath("..", "..", "reo", "src", "reopt_xpress_model.jl"))
include(joinpath("..", "..", "reo", "src", "reopt.jl"))

[tkf]

What's the return value of api.jl_eval_string(b"""try using PyCall...? If it's non-zero it means there are some errors. Also, you can try running some Julia code with api.jl_eval_string and check if libjulia is working.

[NLaws]

Yeah somehow the image is not getting PyCall installed:

>>> import os
>>> from julia.api import LibJulia
>>> julia_img_file = os.path.join("julia_envs", "Xpress", "JuliaXpressSysimage.so")
>>> api = LibJulia.load()
>>> api.sysimage = julia_img_file
>>> api.init_julia()
[ Info: Xpress: Found license file /home/nlaws/xpressmp/bin/xpauth.xpr
[ Info: Xpress: Development license detected.
>>> api.jl_eval_string(b"""try using PyCall; catch err; @error "using PyCall" exception = (err, catch_backtrace()); end""")
┌ Error: using PyCall
│   exception =
│    ArgumentError: Package PyCall not found in current path:
│    - Run `import Pkg; Pkg.add("PyCall")` to install the PyCall package.
│    
│    Stacktrace:
│     [1] require(::Module, ::Symbol) at ./loading.jl:892
│     [2] top-level scope at none:1
└ @ Main none:1
139991732824160
>>> 

[tkf]

Do you have PyCall in the default environment? It might be that PyCall already exists in the image but the package loader still checks Project.toml to see if it is "allowed" to use PyCall.

(PyJulia probably should use Base.require to avoid this error.)

[NLaws]

No PyCall is not in the default julia environment - I was wondering about that. The odd thing is that this process worked on the Linux server before I added the include statements to the precompile.jl file and added the precompile_execution_file input to PackageCompiler.create_sysimage. Before that I was just includeing the precompile.jl file in the build_julia_image function, and precompile.jl only had using statements in it.

[NLaws]

and yes the image does have PyCall installed:

$ julia -J JuliaXpressSysimage.so 
[ Info: Xpress: Found license file /home/nlaws/xpressmp/bin/xpauth.xpr
[ Info: Xpress: Development license detected.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.4) pkg> activate .
 Activating environment at `/julia_envs/Xpress/Project.toml`

(Xpress) pkg> st
Status `/julia_envs/Xpress/Project.toml`
  [39de3d68] AxisArrays v0.4.3
  [4076af6c] JuMP v0.21.2
  [b8f27783] MathOptInterface v0.9.14
  [d8a4904e] MutableArithmetics v0.2.9
  [9b87118b] PackageCompiler v1.1.1
  [438e738f] PyCall v1.91.4
  [9e70acf3] Xpress v0.11.0 #master (https://github.com/JuliaOpt/Xpress.jl.git)

[NLaws]

@tkf here is what I get from api.jl_eval_string

>>> api.jl_eval_string("7*3")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

[tkf]

You need to pass bytes: api.jl_eval_string(b"7*3")

But this problem is already solved, right? For now [*1], you need to add PyCall to the global environment. Alternatively, you can set the environment variable JULIA_PROJECT.

[*1] Until I implement a more sneaky way to import PyCall.

[NLaws]

Ah sorry about that:

>>> api.jl_eval_string(b"7*3")
Segmentation fault (core dumped)

Not sure what's going on there?

I'll try setting the JULIA_PROJECT variable (unfortunately cannot mess with the global environment due to security requirements on this server).

I'm also piece-by-piece pulling out pieces of the system image build process to see what led to the problem. (Removing the include statements from precompile.jl did not help.)

[tkf]

Hmm... That segmentation fault is strange.

[NLaws]

@tkf setting JULIA_PROJECT fixed the issue! Thank you for your timely help! And everything that you've done with PyJulia!