Executable `polymake_run_script` loads system `libperl` but is compiled with `Perl_jll`

I encounter this problem when precompiling Polymake.jl, which invokes polymake_run_script and then fails, because system perl on Fedora does not define PL_exit_flags. The following "fix" illustrates the problem:

mv polymake_run_script polymake_run_script_ORIG Then I create a script that overwrite the environment variable: cat polymake_run_script LD_LIBRARY_PATH=/home/XXX/.julia/artifacts/06ceadfe6189348f596024106c6a86d1d0ae2273/lib:$LD_LIBRARY_PATH /home/XXX/.julia/artifacts/ed052e20a0e8679067418195e52f8837b27e7a7e/bin/polymake_run_script_ORIG $@ This makes Polymake.jl precompile.

I should mention that gmp, mpfr etc. also point to system libraries, but these define all symbols.

I'm not so much of an expert on JLLWrapper, but is static linking an option?

Thanks for the report, unfortunately the JLLWrappers do not really play well with system-wide julia installations (note: using system libraries for julia itself is not really supported by julia). I don't think static linking is really an option here (due to the way binarybuilder works and for polymake as well). Nonetheless, it would be great if you could provide some more details on the setup, maybe we can find another way to avoid this error:

julia version versioninfo()
package versions ]status -m
original LD_LIBRARY_PATH: you could add echo $LD_LIBRARY_PATH >> /tmp/polymake_lib.log to the wrapper
output from ldd /path/to/polymake_run_script_ORIG >> /tmp/polymake_lib.log before adding the extra path (or lddtree if you have that)

The path you are adding should already be in the LD_LIBRARY_PATH but maybe it is hidden behind something like /usr/lib64.

Thanks for taking this up so quickly. You were asking the exact right question. [And thanks also for mentioning lddtree, I didn't know of it.] I didn't even see in the JLLWrapper code that LD_LIBRARY_PATH was adjusted, or does this happen in the Polymake scripts? In any case, a lot of artifact paths show up, but the first paths are system ones. My guess would be that the system path comes from some package in the dependency chain that is system-wide. Does that ring a bell?

julia version versioninfo()

Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: AMD Ryzen 9 3900XT 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, znver2)

package versions ]status -m

      Status `~/.julia/environments/polymake/Manifest.toml`
  [1f15a43c] CxxWrap v0.11.2
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.1
  [1914dd2f] MacroTools v0.5.6
  [4fe8b98c] Mongoc v0.6.2
  [69de0a69] Parsers v1.1.0
  [d720cf60] Polymake v0.5.5
  [21216c6a] Preferences v1.2.1
  [e134572f] FLINT_jll v200.700.100+0
  [78b55507] Gettext_jll v0.20.1+7
  [dd4b983a] LZO_jll v2.10.0+3
  [94ce4f54] Libiconv_jll v1.16.0+7
  [90100e71] MongoC_jll v1.16.2+1
  [68e3532b] Ncurses_jll v6.1.0+5
  [76642167] Ninja_jll v1.10.2+0
  [458c3c95] OpenSSL_jll v1.1.1+6
  [80dd9cbb] PPL_jll v1.2.0+1
  [83958c19] Perl_jll v5.30.3+2
  [05236dd9] Readline_jll v8.0.4+1
  [02c8fc9c] XML2_jll v2.9.11+0
  [3161d3a3] Zstd_jll v1.4.8+0
  [508c9074] bliss_jll v0.73.0+1
  [28df3c45] boost_jll v1.71.0+3
  [f07e07eb] cddlib_jll v0.94.10+1
  [3eaa8342] libcxxwrap_julia_jll v0.8.6+0
  [4d8266f6] libpolymake_julia_jll v0.4.105+0
  [3873f7d0] lrslib_jll v0.2.0+0
  [55c6dc9b] nauty_jll v2.6.12+1
  [6690c6e9] normaliz_jll v300.800.901+0
  [7c209550] polymake_jll v400.300.1+0
  [fe1e1685] snappy_jll v1.1.8+1
  [0dad84c5] ArgTools
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads
  [b77e0a4c] InteractiveUtils
  [b27032c2] LibCURL
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [fa267f1f] TOML
  [a4e569a6] Tar
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll
  [781609d7] GMP_jll
  [deac9b47] LibCURL_jll
  [29816b5a] LibSSH2_jll
  [3a97d323] MPFR_jll
  [c8ffd9c3] MbedTLS_jll
  [14a3606d] MozillaCACerts_jll
  [83775a58] Zlib_jll
  [8e850ede] nghttp2_jll
  [3f19e933] p7zip_jll

original LD_LIBRARY_PATH: you could add echo $LD_LIBRARY_PATH >> /tmp/polymake_lib.log to the wrapper

/usr/bin/../lib64/julia:/home/martin/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/lib:/home/martin/.julia/artifacts/aef720c6389e0e2b15462757cbcc370d4f892d98/lib:/home/martin/.julia/artifacts/f2662cbb8a75e1b8f1d05e87781a8ca83702f5ed/lib:/home/martin/.julia/artifacts/473dc2ed3d46252c87733b7fdca3da64c6f9f917/lib:/lib64:/home/martin/.julia/artifacts/c91b35452c8fa91e8ac43993a2bc8fb4f78476bb/lib:/home/martin/.julia/artifacts/93d795500c7b63f53b10478c17435f32e7a3c443/lib:/home/martin/.julia/artifacts/b801935f313061d64e28d8a28c39cdc24f139920/lib:/home/martin/.julia/artifacts/502dc6101ca029abe687e905f3f5f75b2bdd5254/lib:/home/martin/.julia/artifacts/06ceadfe6189348f596024106c6a86d1d0ae2273/lib:/home/martin/.julia/artifacts/adae6b9e6ad8b51d5dbfdb27d0cbe200e68e6d42/lib:/home/martin/.julia/artifacts/9960e060b732618174ffd95804d6635833f99513/lib:/home/martin/.julia/artifacts/06e6732c6f448b09033916da99c20057a0d7d395/lib:/home/martin/.julia/artifacts/8d0503febedd2202cb73e7696cc874a6cf68129c/lib:/home/martin/.julia/artifacts/73247ee194631a1a0100b72573cea7adadb6d3bd/lib:/home/martin/.julia/artifacts/59fdf871700afeca91caa706b8c4baf6e654dc3a/lib:/home/martin/.julia/artifacts/1573202751ea069d56c245ae168d2937f43a29ec/lib:/home/martin/.julia/artifacts/ed052e20a0e8679067418195e52f8837b27e7a7e/lib:/usr/bin/../lib64/julia:/usr/bin/../lib64

output from ldd /path/to/polymake_run_script_ORIG >> /tmp/polymake_lib.log before adding the extra path (or lddtree if you have that)

polymake_run_script_BIN => /home/martin/.julia/artifacts/ed052e20a0e8679067418195e52f8837b27e7a7e/bin/polymake_run_script_BIN (interpreter => /lib64/ld-linux-x86-64.so.2)
    libpolymake.so.4.3 => /home/martin/.julia/artifacts/1573202751ea069d56c245ae168d2937f43a29ec/lib/libpolymake.so.4.3
        libperl.so => /lib64/libperl.so
            libcrypt.so.2 => /lib64/libcrypt.so.2
    libpolymake-apps-rt.so.4.3 => /home/martin/.julia/artifacts/1573202751ea069d56c245ae168d2937f43a29ec/lib/libpolymake-apps-rt.so.4.3
    libflint.so.15 => /home/martin/.julia/artifacts/aef720c6389e0e2b15462757cbcc370d4f892d98/lib/libflint.so.15
        ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
    libdl.so.2 => /lib64/libdl.so.2
    libmpfr.so.6 => /lib64/libmpfr.so.6
    libgmp.so.10 => /lib64/libgmp.so.10
    libpthread.so.0 => /lib64/libpthread.so.0
    libstdc++.so.6 => /usr/bin/../lib64/julia/libstdc++.so.6
    libm.so.6 => /lib64/libm.so.6
    libgomp.so.1 => /usr/bin/../lib64/julia/libgomp.so.1
    libgcc_s.so.1 => /usr/bin/../lib64/julia/libgcc_s.so.1
    libc.so.6 => /lib64/libc.so.6

For each executable product the JLLWrappers create a wrapper function that automatically adjusts the LD_LIBRARY_PATH using the LIBPATH_list entries for all the dependencies, see https://github.com/JuliaPackaging/JLLWrappers.jl/blob/master/src/products/executable_generators.jl .

Splitting that long line gives:

/usr/bin/../lib64/julia
/home/martin/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/lib
/home/martin/.julia/artifacts/aef720c6389e0e2b15462757cbcc370d4f892d98/lib
/home/martin/.julia/artifacts/f2662cbb8a75e1b8f1d05e87781a8ca83702f5ed/lib
/home/martin/.julia/artifacts/473dc2ed3d46252c87733b7fdca3da64c6f9f917/lib
/lib64
/home/martin/.julia/artifacts/c91b35452c8fa91e8ac43993a2bc8fb4f78476bb/lib
/home/martin/.julia/artifacts/93d795500c7b63f53b10478c17435f32e7a3c443/lib
/home/martin/.julia/artifacts/b801935f313061d64e28d8a28c39cdc24f139920/lib
/home/martin/.julia/artifacts/502dc6101ca029abe687e905f3f5f75b2bdd5254/lib
/home/martin/.julia/artifacts/06ceadfe6189348f596024106c6a86d1d0ae2273/lib
/home/martin/.julia/artifacts/adae6b9e6ad8b51d5dbfdb27d0cbe200e68e6d42/lib
/home/martin/.julia/artifacts/9960e060b732618174ffd95804d6635833f99513/lib
/home/martin/.julia/artifacts/06e6732c6f448b09033916da99c20057a0d7d395/lib
/home/martin/.julia/artifacts/8d0503febedd2202cb73e7696cc874a6cf68129c/lib
/home/martin/.julia/artifacts/73247ee194631a1a0100b72573cea7adadb6d3bd/lib
/home/martin/.julia/artifacts/59fdf871700afeca91caa706b8c4baf6e654dc3a/lib
/home/martin/.julia/artifacts/1573202751ea069d56c245ae168d2937f43a29ec/lib
/home/martin/.julia/artifacts/ed052e20a0e8679067418195e52f8837b27e7a7e/lib
/usr/bin/../lib64/julia
/usr/bin/../lib64

The paths at the beginning should be fine as there is no libperl inside the julia subdirectory, but the problematic one is the /lib64 which seems to come from the stdlib Zlib_jll, you can check with using Zlib_jll; Zlib_jll.LIBPATH_list. The way this path is generated is not really straightforward: Usually julia will load most dependencies explicitly using dlopen to make sure they are opened with the correct (julia-specific) path, which should be /usr/bin/../lib64/julia in your case (you can have a look at the paths with using Libdl; dllist()). The Zlib_jll wrapper will use the path from which libz.so (Zlib_jll) was loaded to fill that LIBPATH_list. There should be mostly symlinks for all system-libraries in there, curiously libz.so seems to be missing there in the fedora package, but adding that symlink wont help.

It looks like that on fedora some other system-library that julia needs (at least one of libssh2, libgit2 or libcurl) has a dependency on libz.so which the corresponding one that julia would normally ship doesn't have. Then libz.so is loaded automatically as a dependency (instead of via dlopen) from the default path and thus from /lib64. And this way we end up with that system path in LD_LIBRARY_PATH which will override all the paths that come later in the list with the corresponding system libraries (if they exist).

There might be some ways to work around this, and maybe also the stdlibs need some adjustments but I need some more time to think about this. (I think there is some additional stdlib/jll-refactoring planned for julia 1.7 anyway)

This should not happen anymore with Polymake.jl >= 0.6.2 as we removed that call to polymake_run_script. (It is now executed when the jll is built in Yggdrasil and the resulting json files are shipped with the jll)

oscar-system / libpolymake-julia

Executable `polymake_run_script` loads system `libperl` but is compiled with `Perl_jll` #17