Could not load python library when run from compiled julia app on another machine

[sairus7]

When I compile my app that has a PyCall dependency, it could not find python library, because it is compiled with absolute path to that library found on dev machine. So when I run my app on another machine, it can't find it:

fatal: error thrown and no exception handler available.
InitError(mod=:PyCall, error=ErrorException("could not load library "C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll"
The specified module could not be found. "))
jl_errorf at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:77
jl_load_dynamic_library at /cygdrive/c/buildbot/worker/package_win64/build/src\dlload.c:284
#dlopen#3 at .\libdl.jl:117
dlopen at .\libdl.jl:117 [inlined]
init at C:\absolute\path\from\another\machine\.julia\packages\PyCall\7a7w0\src\pyinit.jl:149
jfptr___init___58920.clone_1 at C:\_KTAuto\backend\MainBackendCompiled\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:73

So its this line pyinit.jl:149: https://github.com/JuliaPy/PyCall.jl/blob/6b18f387e54b6d5a31b1dd6e65a26c45471c2356/src/pyinit.jl#L150

[stevengj]

Yes, compiling in the libpython into PyCall was something we did to improve load times (#169). Nowadays, it might be possible to revisit this; see e.g. the PythonCall.jl package for a more dynamic approach.

[sairus7]

There are some other problems compiling app with PythonCall - see here: https://github.com/cjdoris/PythonCall.jl/issues/146

@stevengj Can you please explain a little bit more, what should be fixed in PyCall to exclude fixed library paths from compiled binaries? Or any other ways to use it with compiled app?

[efJerryYang]

I also encountered this problem today, almost the exact same error message as @sairus7 . At first I used create_sysimage it crashed in a very similar way, so I chose create_app but got this error.

[stevengj]

I wonder if it's as simple as inserting the following line here:

libpython = relpath.(libpython, @__DIR__)

Want to give it a try?

[efJerryYang]

Oh, things seem to crash in the same way with fixed absolute path... Any other operations needed before building the app?

And I wonder if this is the right absolute path for build.jl? I build my app in default julia environment. @stevengj

C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl

[sairus7]

And I wonder if this is the right absolute path for build.jl?

Make sure you dev PyCall in your project environment and make changes into .julia/dev/PyCall/deps/build.jl

[efJerryYang]

Thanks, but this error was thrown when building the pkgs in my project environment

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -y numpy` in root environment
ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)
Closest candidates are:
  relpath(!Matched::String, ::String) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:536
  relpath(!Matched::AbstractString, ::AbstractString) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:573
Stacktrace:
 [1] _broadcast_getindex_evalf
   @ .\broadcast.jl:670 [inlined]
 [2] _broadcast_getindex
   @ .\broadcast.jl:643 [inlined]
 [3] getindex
   @ .\broadcast.jl:597 [inlined]
 [4] copy
   @ .\broadcast.jl:875 [inlined]
 [5] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(relpath), Tuple{Base.RefValue{Ptr{Nothing}}, Base.RefValue{String}}})
   @ Base.Broadcast .\broadcast.jl:860
 [6] top-level scope
   @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:84
 [7] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [8] top-level scope
   @ none:5
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:43

[sairus7]

I build my app in default julia environment.

To compile the project into an app I use project own environment, not the default one. This is typical script compile_app.sh, notice --project=@. argument:

julia -e 'using Pkg; Pkg.add("PackageCompiler")'

julia --project=@. --startup-file=no -e 'using Pkg; Pkg.instantiate()'

julia --project=@. --startup-file=no -e '
using PackageCompiler;
PackageCompiler.create_app(pwd(), "MyProjectCompiled";
    cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)",
    include_transitive_dependencies=false,
    filter_stdlibs=true,
    precompile_execution_file=["test/runtests.jl"])
'

[sairus7]

ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)

Maybe that's because libpython === nothing?

[efJerryYang]

Sorry, is this solution works for you? And that may be my problem...

[sairus7]

I will check this in a couple of hours

[efJerryYang]

Great, but problem is that the path actually should not be nothing, and after I commented that line pkgs could be built without error (in the Demo environment)

[efJerryYang]

I confirmed that the error was caused by that line of code.

after commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
Precompiling project...
  5 dependencies successfully precompiled in 43 seconds (235 already precompiled)

cancel commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
...

[efJerryYang]

I got the correct build result after replace libpython with libpy_name like this, which is really strange to assign a pointer type with string value.

libpython = relpath.(libpy_name, @__DIR__)

The value of libpy_name is just the fixed absolute path C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll

[sairus7]

libpython = relpath.(libpython, @__DIR__)

This fix didn't work for me - it runs through tests and compilation stages, but trying to run the app gives the same error as in the first message.

[efJerryYang]

Hi, I was wrong in the last comment. I stuck here when replaced both libpython with libpy_name, but encountered an error when loading directory in startup.jl.

julia> import Pkg; Pkg.precompile()
Precompiling project...
  ✗ PyCall
  0 dependencies successfully precompiled in 2 seconds (239 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]

Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to C:\Users\JerryYang\.julia\compiled\v1.7\PyCall\jl_DD83.tmp.
ERROR: LoadError: could not load library "..\..\..\..\conda\3\python39.dll"
The specified module could not be found. . Please run `Pkg.build("PyCall")` if your Python build has changed
Stacktrace:
  [1] error(::String, ::String)
    @ Base .\error.jl:42
  [2] top-level scope
    @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\startup.jl:51
  [3] include(mod::Module, _path::String)
    @ Base .\Base.jl:418
  [4] include(x::String)
    @ PyCall C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:1
  [5] top-level scope
    @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:38
  [6] include
    @ .\Base.jl:418 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1318
  [8] top-level scope
    @ none:1
  [9] eval
    @ .\boot.jl:373 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [11] top-level scope
    @ none:1
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\startup.jl:41
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:1

I look into it, and it is this line startup.jl:48 just the same as you found at the very first:

https://github.com/JuliaPy/PyCall.jl/blob/6b18f387e54b6d5a31b1dd6e65a26c45471c2356/src/startup.jl#L48

[efJerryYang]

it runs through tests and compilation stages, but trying to run the app gives the same error as in the first message.

It actually did not run through tests, with a skipped error message like this:

Precompiling project...
  3 dependencies successfully precompiled in 40 seconds (236 already precompiled, 1 skipped during auto due to previous errors)

[efJerryYang]

I think just replace the absolute libpy_name with its related path representation is not enough, because it is libpython that caused those errors. There may need some more changes to be made in related procedure, even not just the file build.jl.

[efJerryYang]

Such modification only changed the variable libpython stored in deps.jl, you can see here:

const python = "C:\\Users\\JerryYang\\.julia\\conda\\3\\python.exe"
const libpython = "..\\..\\..\\..\\conda\\3\\python39.dll"
const pyprogramname = "C:\\Users\\JerryYang\\.julia\\conda\\3\\python.exe"
const pyversion_build = v"3.9.10"
const PYTHONHOME = "C:\\Users\\JerryYang\\.julia\\conda\\3"

"True if we are using the Python distribution in the Conda package."
const conda = true