Can't get HDF5.jl work with Julia running in docker (julia:1.8-alpine3.17) - can't find libmpi.so.12

[dhanak]

I'm trying to get HDF5.jl working with docker based Julia, but I can't get past this error message:

ERROR: LoadError: InitError: could not load library "/root/.julia/artifacts/3ff8e041740c57fdef9e487eed98ac8f3c1a70ad/lib/libhdf5.so"
Error loading shared library libmpi.so.12: No such file or directory (needed by /root/.julia/artifacts/3ff8e041740c57fdef9e487eed98ac8f3c1a70ad/lib/libhdf5.so)

To reproduce, run the following command:

$ docker run --rm julia:1.8-alpine3.17 julia -e 'run(`apk add libsz`); using Pkg; Pkg.add("HDF5"); Pkg.precompile()'

Locally, on a Debian system, everything works smoothly. I checked the insides of the docker container, and libmpi.so is there as part of some artifact (probably downloaded with MPICH_jll, I'm not entirely sure).

[mkitti]

I recommend setting the environment variable LD_DEBUG to all or libs.

https://bnikolic.co.uk/blog/linux-ld-debug.html

[dhanak]

I recommend setting the environment variable LD_DEBUG to all or libs.

https://bnikolic.co.uk/blog/linux-ld-debug.html

Out of luck: the LD_DEBUG envvar doesn't seem to do anything in this docker image.

[mkitti]

Umm... how are you setting it?

[dhanak]

$ docker run --rm julia:1.8-alpine3.17 env LD_DEBUG=libs julia -e 'run(`apk add libsz`); using Pkg; Pkg.add("HDF5"); Pkg.precompile()'

Note that the same technique works well for debian based images (albeit there is no issue with the HDF5 library there, it doesn't even try to load the libmpi.so library):

$ docker run --rm julia:1.8-bullseye env LD_DEBUG=libs julia -e 'using Pkg; Pkg.add("HDF5"); Pkg.precompile()'

This works just fine, and dumps a ton of dynlib loading info in the process.

[mkitti]

Mose Giordano says

the bug is very likely related to https://github.com/JuliaLang/julia/issues/40556

[dhanak]

Reading through that issue, I'm certain that it is the same issue. The “missing” libmpi.so.12 is downloaded as an artifact of the dependent MPICH_jll library, so it is in a different directory than libhdf5.so, which depends on it.

It's a shame this issue hasn't been solved since 2021, it must be a real pain for Julia users on Alpine.

Duplicate of JuliaLang/julia#40556. (Relinking so that it becomes a proper mention with back-reference.)