search

JuliaIO / HDF5.jl

Save and load data in the HDF5 file format from Julia
https://juliaio.github.io/HDF5.jl
MIT License
380 stars 138 forks source link

Changed requirements in HDF5_jll's `libhdf5.so` for `libcurl.so`? #1117

Open sloede opened 10 months ago

sloede commented 10 months ago

Up until recently (say 1-2 weeks ago), we always had to set

LD_PRELOAD=/lib/x86_64-linux-gnu/libcurl.so.4

in our GitHub Action workflow when using the locally installed /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so in the GitHub runner. Otherwise, we would get an error such as

/opt/hostedtoolcache/julia/1.9.3/x64/bin/../lib/julia/libcurl.so: version `CURL_OPENSSL_4' not found (required by /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so)

(see, e.g., here).

This LD_PRELOAD funny business was OK-ish, since it was only required for CI jobs on GitHub runners and it also worked with the JLL-provided libhdf5.so library.

However, since a few days ago, preloading the system libcurl.so does not work anymore with the JLL-provided libhdf5.so. If we do that, we get the following error:

/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_4' not found (required by /home/runner/.julia/artifacts/2829a1f6a9ca59e5b9b53f52fa6519da9c9fd7d3/lib/libhdf5.so)

(see, e.g., here).

Does anyone know where this change is coming from and/or whether there is a way around this? The trouble this causes us is that in our CI scripts, we now have to install HDF5 without LD_PRELOAD being set, then configure our project to use the local libhdf5.so, then set LD_PRELOAD, then disable it again when we run some Julia tests (which use the JLL libraries), and then set it again once we run our parallel tests with the local MPI+HDF5 stack.

cc @bgeihe

lj-cug commented 10 months ago

I have also encountered this problem. I cannot solve it now.

mkitti commented 10 months ago

Can you narrow down the version numbers are involved?

Does this occur with any version of HDF5.jl? What if you try HDF5.jl 0.16.15?

What versions of HDF5_jll does this occur for?

Libcurl is needed by HDF5 for the the read-only S3 (ROS3) driver.

lj-cug commented 10 months ago

After using lower version JDF5.jl, this problem disappeared. Thank you!

mkitti commented 10 months ago

Can you tell us what versions resolved the issue?

lj-cug commented 10 months ago

At least, I use "add HDF5@0.16.14", there is no problem. If I use "add HDF5", the newest version was used, the above issue happened. The reason is that the Libcurl library version is low, which was not suitable for julia. I don't understand the "read-only S3 (ROS3) driver" for HDF5. I always use HDF5 in my C or FORTRAN codes. This is the first time to use HDF5 in julia.

mkitti commented 10 months ago

Basically, the ROS3 driver let's the HDF5 library talk to a HDF5 file stored in Amazon S3. This requires libcurl.

I'm guessing that the operating system libcurl uses OpenSSL and thus uses symbol CURL_OPENSSL_4 rather than CURL_4

lj-cug commented 10 months ago

That's right! The error printout shows me the libcurl uses OpenSSL. Should I install lincurl using other SSL configuration? Maybe this can solve the problem.

mkitti commented 10 months ago

Julia's libcurl recipe is here: https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/LibCURL/common.jl

It usually uses mbedTLS rather than OpenSSL.