search

JuliaPy / pyjulia

python interface to julia
MIT License
884 stars 101 forks source link

PyCall 1.92.0 breaks sysimage build? #424

Closed grahamgill closed 4 years ago

grahamgill commented 4 years ago

I see PyCall 1.92.0 was just released.

Now my system image build, of the pyjulia system image on top of a custom sysimage, is failing. See the output.

I've tried under both Julia 1.4.2 and 1.5.2, Python 3.8.5, Pyjulia 0.5.6.

Actually building on top of the standard julia sysimage breaks also, with the same errors produced, using build_sysimage("lib/test.so", compiler_env=".") and build_sysimage("lib/test.so").

When PyCall 1.91.4 was the latest version, that was used, and the build succeeded, although with warnings about "incremental precompilation of this module may be fatally broken" or similar when I was building the pyjulia sysimage on top of the custom sysimage.

Can I control the version of PyCall that is pulled in to make a sysimage? Can I use an existing PyCall installation rather than always downloading a version to the temporary environment?

At worst I can simply not do the pyjulia sysimage build on top of the custom sysimage, since using the custom sysimage directly with e.g. jl = Julia(sysimage="lib/rr_sys.so") seems to work, just giving a moderately slower startup time.

> python
Python 3.8.5 (default, Sep  5 2020, 10:50:12) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia.sysimage import build_sysimage
>>> build_sysimage("lib/pj_rr_sys.so", base_sysimage="lib/rr_sys.so", compiler_env=".") 
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
[ Info: Loading PackageCompiler...
[ Info: Installing PyCall...
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
Updating `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
  [438e738f] + PyCall v1.92.0
Updating `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Manifest.toml`
  [8f4d0f93] + Conda v1.4.1
  [682c06a0] + JSON v0.21.1
  [1914dd2f] + MacroTools v0.5.5
  [69de0a69] + Parsers v1.0.10
  [438e738f] + PyCall v1.92.0
  [81def892] + VersionParsing v1.2.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [8dfed614] + Test
  [4ec0a83e] + Unicode
[ Info: Compiling a temporary system image without `PyCall`...
[ Info: PackageCompiler: creating system image object file, this might take a while...
[ Info: Compiling system image...
ERROR: LoadError: LoadError: LoadError: UndefVarError: pyversion_build not defined
Stacktrace:
 [1] top-level scope at none:1
 [2] eval(::Module, ::Any) at ./boot.jl:331
 [3] @static(::LineNumberNode, ::Module, ::Any) at ./osutils.jl:19
 [4] #macroexpand#36 at ./expr.jl:108 [inlined]
 [5] macroexpand at ./expr.jl:107 [inlined]
 [6] docm(::LineNumberNode, ::Module, ::Any, ::Any, ::Bool) at ./docs/Docs.jl:521 (repeats 2 times)
 [7] @doc(::LineNumberNode, ::Module, ::String, ::Vararg{Any,N} where N) at ./boot.jl:464
 [8] include(::Function, ::Module, ::String) at ./Base.jl:380
 [9] include(::Module, ::String) at ./Base.jl:368
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:924
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:913
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:913
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
ERROR: LoadError: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/graham/.julia/compiled/v1.5/PyCall/mmQEj_RLR2M.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923
 [6] include(::Function, ::Module, ::String) at ./Base.jl:380
 [7] include(::Module, ::String) at ./Base.jl:368
 [8] exec_options(::Base.JLOptions) at ./client.jl:296
 [9] _start() at ./client.jl:506
 [10] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [11] invokelatest(::Any) at ./essentials.jl:709
 [12] top-level scope at none:1
in expression starting at /home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl:1
ERROR: LoadError: failed process: Process(`/home/graham/.local/bin/julia-py --color=yes --startup-file=no --sysimage=/tmp/jl_ezKvE4/pj_rr_sys.so --project=. --compile=all --trace-compile=/tmp/jl_xRTRH4 /home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] read(::Cmd) at ./process.jl:412
 [3] run_precompilation_script(::String, ::String, ::String) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:177
 [4] create_sysimg_object_file(::String, ::Array{String,1}; project::String, base_sysimage::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, cpu_target::String, script::Nothing, isapp::Bool) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:204
 [5] create_sysimage(::Array{Symbol,1}; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::String, isapp::Bool) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:423
 [6] (::var"#7#8")(::String) at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:34
 [7] mktempdir(::var"#7#8", ::String; prefix::String) at ./file.jl:682
 [8] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [9] top-level scope at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:24
 [10] include(::Function, ::Module, ::String) at ./Base.jl:380
 [11] include(::Module, ::String) at ./Base.jl:368
 [12] exec_options(::Base.JLOptions) at ./client.jl:296
 [13] _start() at ./client.jl:506
 [14] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [15] invokelatest(::Any) at ./essentials.jl:709
 [16] top-level scope at none:1
in expression starting at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:23
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/graham/.local/lib/python3.8/site-packages/julia/sysimage.py", line 110, in build_sysimage
    check_call(build_sysimage_cmd(julia_py, julia, compile_args), cwd=path)
  File "/home/graham/.local/lib/python3.8/site-packages/julia/sysimage.py", line 64, in check_call
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/graham/.local/bin/julia-py', '--julia', 'julia', '--compiled-modules=no', '--color=yes', '/home/graham/.local/lib/python3.8/site-packages/julia/compile.jl', '.', '/home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl', '/home/graham/projects/julia/RecommendRebalance/lib/pj_rr_sys.so', '/home/graham/projects/julia/RecommendRebalance/lib/rr_sys.so']' returned non-zero exit status 1.
>>>
tkf commented 4 years ago

Thanks for filing the issue. I think https://github.com/JuliaPy/PyCall.jl/pull/841 should fix this.

grahamgill commented 4 years ago

Can I control the version of PyCall that is pulled in to make a sysimage? Can I use an existing PyCall installation rather than always downloading a version to the temporary environment?

@tkf Would it be feasible to allow pulling a Project and/or Manifest, if supplied, into the temporary environment used for creating a sysimage including PyCall, and then Pkg.adding or Pkg.instantiateing to control versions of PyCall and its dependencies? We have a mixed python and julia project with a build process we'd like to be able to lock down. If you think that's a reasonable approach I would take a stab at it.

tkf commented 4 years ago

Yeah, I agree that's a great idea. We'll have it once #422 is merged.

tkf commented 4 years ago

Ref https://github.com/JuliaRegistries/General/pull/22235

grahamgill commented 4 years ago

Thanks @tkf. The custom sysimage builds for me with PyCall 1.92.1 and is usable in the context of the julia module in python.

Now I'm back to a set of warnings when building the custom sysimage that look like the following. This is with Julia 1.4.2. Is this considered normal?

>>> build_sysimage("lib/pj_rr_sys.so", base_sysimage="lib/rr_sys.so", compiler_env=".") 
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.9cemj3bm/Project.toml`
[ Info: Loading PackageCompiler...
[ Info: Installing PyCall...
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.9cemj3bm/Project.toml`
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
  Installed PyCall ─ v1.92.1
   Updating `/tmp/tmp.pyjulia.sysimage.9cemj3bm/Project.toml`
  [438e738f] + PyCall v1.92.1
   Updating `/tmp/tmp.pyjulia.sysimage.9cemj3bm/Manifest.toml`
  [8f4d0f93] + Conda v1.4.1
  [682c06a0] + JSON v0.21.1
  [1914dd2f] + MacroTools v0.5.5
  [69de0a69] + Parsers v1.0.10
  [438e738f] + PyCall v1.92.1
  [81def892] + VersionParsing v1.2.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [de0858da] + Printf 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [4ec0a83e] + Unicode 
   Building PyCall → `~/.julia/packages/PyCall/BcTLp/deps/build.log`
[ Info: Compiling system image...
WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):33 =#, Expr(:function, Expr(:call, :package_slug, Expr(:::, :uuid, :UUID), Expr(:kw, Expr(:::, :p, :Int), 5)), Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):34 =#, :crc = Expr(:call, :_crc32c, :uuid), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):35 =#, :crc = Expr(:call, :_crc32c, "/home/graham/.local/bin/julia-py", :crc), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):36 =#, Expr(:return, Expr(:call, :slug, :crc, :p)))))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID, Int64) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, Expr(:call, :julia_exename) = Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, "/home/graham/.local/bin/julia-py"))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition julia_exename() in module Base at util.jl:497 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:52.
  ** incremental compilation may be fatally broken for this module **

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
│   exception = Required dependency MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09] failed to load from a cache file.
└ @ Base loading.jl:1041
WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):33 =#, Expr(:function, Expr(:call, :package_slug, Expr(:::, :uuid, :UUID), Expr(:kw, Expr(:::, :p, :Int), 5)), Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):34 =#, :crc = Expr(:call, :_crc32c, :uuid), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):35 =#, :crc = Expr(:call, :_crc32c, "/home/graham/.local/bin/julia-py", :crc), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):36 =#, Expr(:return, Expr(:call, :slug, :crc, :p)))))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID, Int64) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, Expr(:call, :julia_exename) = Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, "/home/graham/.local/bin/julia-py"))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition julia_exename() in module Base at util.jl:497 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:52.
  ** incremental compilation may be fatally broken for this module **

┌ Warning: Module VersionParsing with build ID 23786452100182 is missing from the cache.
│ This may mean VersionParsing [81def892-9a0e-5fdd-b105-ffc91e053289] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1016
WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):33 =#, Expr(:function, Expr(:call, :package_slug, Expr(:::, :uuid, :UUID), Expr(:kw, Expr(:::, :p, :Int), 5)), Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):34 =#, :crc = Expr(:call, :_crc32c, :uuid), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):35 =#, :crc = Expr(:call, :_crc32c, "/home/graham/.local/bin/julia-py", :crc), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):36 =#, Expr(:return, Expr(:call, :slug, :crc, :p)))))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID, Int64) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, Expr(:call, :julia_exename) = Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, "/home/graham/.local/bin/julia-py"))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition julia_exename() in module Base at util.jl:497 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:52.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):33 =#, Expr(:function, Expr(:call, :package_slug, Expr(:::, :uuid, :UUID), Expr(:kw, Expr(:::, :p, :Int), 5)), Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):34 =#, :crc = Expr(:call, :_crc32c, :uuid), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):35 =#, :crc = Expr(:call, :_crc32c, "/home/graham/.local/bin/julia-py", :crc), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):36 =#, Expr(:return, Expr(:call, :slug, :crc, :p)))))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID, Int64) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, Expr(:call, :julia_exename) = Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, "/home/graham/.local/bin/julia-py"))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition julia_exename() in module Base at util.jl:497 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:52.
  ** incremental compilation may be fatally broken for this module **

┌ Warning: Module MacroTools with build ID 23803869645542 is missing from the cache.
│ This may mean MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1016
WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):33 =#, Expr(:function, Expr(:call, :package_slug, Expr(:::, :uuid, :UUID), Expr(:kw, Expr(:::, :p, :Int), 5)), Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):34 =#, :crc = Expr(:call, :_crc32c, :uuid), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):35 =#, :crc = Expr(:call, :_crc32c, "/home/graham/.local/bin/julia-py", :crc), #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):36 =#, Expr(:return, Expr(:call, :slug, :crc, :p)))))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition package_slug(Base.UUID, Int64) in module Base at loading.jl:144 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:34.
  ** incremental compilation may be fatally broken for this module **

WARNING: eval into closed module Base:
Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, Expr(:call, :julia_exename) = Expr(:block, #= Symbol("/home/graham/.local/lib/python3.8/site-packages/julia/patch.jl"):52 =#, "/home/graham/.local/bin/julia-py"))
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition julia_exename() in module Base at util.jl:497 overwritten at /home/graham/.local/lib/python3.8/site-packages/julia/patch.jl:52.
  ** incremental compilation may be fatally broken for this module **

┌ Warning: Module VersionParsing with build ID 23786452100182 is missing from the cache.
│ This may mean VersionParsing [81def892-9a0e-5fdd-b105-ffc91e053289] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1016
[ Info: PackageCompiler: creating system image object file, this might take a while...
[ Info: System image is created at /home/graham/projects/julia/RecommendRebalance/lib/pj_rr_sys.so
>>>
tkf commented 4 years ago

Hmm... I thought I suppressed these warnings (but I'm not sure). These warnings are expected. I think they should be harmless.

grahamgill commented 4 years ago

Thanks @tkf