Get fatal error only when different user executes app

[hannes-p]

Hi, folks.

I've compiled an app using PackageCompiler v1.1.1 on a 64-bit Linux server (w/ Ubuntu 16.04.6 LTS installed). Long story short: I can use the app w/o any issues; however, as soon as another user tries to execute the app, the user gets a fatal error.

I compile the package using

julia-1.4.1 -q --project --color=yes -e "using PackageCompiler; create_app(\"./\", \"${path_dir_out}\")"

as well as

function julia_main()::Cint
    try
        main(ARGS)
    catch
        Base.invokelatest(Base.display_error, Base.catch_stack())
        return 1
    end
    return 0
end

and the required Project.toml and Manifest.toml files. After successfully compiling the app, I can execute it w/o any problems. As mentioned before, as soon as another user executes the binary, he/she gets a fatal error:

fatal: error thrown and no exception handler available.
#<null>
jl_errorf at /buildworker/worker/package_linux64/build/src/rtutils.c:77
jl_load_dynamic_library at /buildworker/worker/package_linux64/build/src/dlload.c:233
#dlopen#3 at /usr2/myuser/bin/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109
dlopen at /usr2/myuser/bin/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109 [inlined]
dlopen at /usr2/myuser/bin/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109
unknown function (ip: 0x7f3ddbbcea32)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
__init__ at /usr2/myuser/.julia/packages/Lz4_jll/3A8BT/src/wrappers/x86_64-linux-gnu.jl:155
jfptr___init___9244 at /data3/myuser/mytool/bin/mytool.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:74
_julia_init at /buildworker/worker/package_linux64/build/src/init.c:788
main at ./bin/mytool (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at ./bin/mytool (unknown line)

I realized that there are some dependencies w.r.t. the installed Julia version:

#dlopen#3 at /usr2/myuser/bin/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109

However, when compiling the app the way mentioned in the manual, there shouldn't be any dependencies left. Isn't it? The app should run on different machines.

At this point I have to say that I only installed Julia in my home directory, not on the server. I.e., the other users don't have access to Julia, but I think they don't need to.

Looking forward to your replies. Thanks a lot in advance.

[KristofferC]

Do you try to open any library in your code or does any dependency do that? It might be that there is an absolute path stored to that library and it will not work when another user then tries to run it.

[hannes-p]

Hi, Kristoffer. Thanks for your reply. As far as I know I don't (try to) open any library (on purpose). I compiled this app on different servers; it always worked w/o any errors as long as I executed it. Consequently, I think that a package might introduce this dependency. In that specific app, I'm only using packages like ArgParse, DelimitedFiles, DSP, HDF5, LinearAlgebra, Logging, Printf, WAV, YAML, and some other basic packages.

1. How can I figure out if a library introduces such dependencies or absolute paths?

In the meanwhile, I've installed Julia 1.4.1 on the server (i.e., a location different to my home directory). I compiled the app w/ the newly installed Julia version, and all of a sudden, everyone can execute the program (of course w/o any errors). I still believe that this dependency exists; however, since I've installed Julia on a server (where all users have access to it), everyone can execute the app w/o any errors. This is only a workaround, though. At the end I'd need apps where I don't have to install Julia on each machine.

2. Shouldn't the artifacts system solve all issues related to dependencies?

[KristofferC]

I still believe that this dependency exists; however, since I've installed Julia on a server (where all users have access to it), everyone can execute the app w/o any errors.

I'm surprised that installing Julia would help anything. The app should not know anything (nor look for) any external julia installs. Does things start fail if you uninstall that Julia? What Julia version did you use for the app when you first tried?

[hannes-p]

Sorry, I wasn't clear enough; my bad. Let me start all over.

1. In all tests I used the same version of Julia, i.e., v1.4.1.
2. In test one I installed Julia in my home directory, which can only be accessed by me.
   • I used this version in order to compile the app on server A.
   • I could successfully execute the app on server A.
   • No one else could execute the app on server A.
3. In test two I installed Julia on server A, so that everyone could access it.
   • I used this installation (and not the one in my home directory) to compile the app on server A.
   • Everyone could successfully execute the app on server A.

If I switch to a different server B, then the issues will be identical.

Does things start fail if you uninstall that Julia?

I haven't tried this yet, but I'm pretty sure that executing the app will fail due to this dependency. I'll try this on a different server B, since people are already using this app on server A.

[hannes-p]

Hi, folks. I did some more research on this issue and realized that packages like HDF5 caused the problem. Let me start at the beginning.

I compiled comprehensive apps on several Linux servers given a local installation of Julia, i.e., only I had access to Julia on these servers. These apps depended on the HDF5 package (among other packages). Interestingly, the compiled app only worked for my user, since I had access to the HDF5 package of my local Julia installation. As soon as someone else tried to execute the app, he/she got an error message due to a missing local HDF5 package (or more specifically, due to a missing Julia installation).

It seems that the HDF5 package wasn't added as an artifact in the compilation. Please correct me if I'm wrong, but this shouldn't happen. Right? I removed this package, recompiled the app, and it worked like a charm afterwards, i.e., everyone could execute the app.

1. Do you have any idea why the HDF5 package wasn't included in the compilation as an artifact? 2. Do you think that the HDF5 package actually caused this issue, since it was pointing to a local library/binary?

[KristofferC]

1. Do you have any idea why the HDF5 package wasn't included in the compilation as an artifact?

It doesn't use the artifact system so how could it? See https://julialang.github.io/PackageCompiler.jl/dev/apps/#Relocatability-1.