search

grimbough / rhdf5

Package providing an interface between HDF5 and R
http://bioconductor.org/packages/rhdf5
59 stars 22 forks source link

Undefined symbol: HMAC #109

Closed brgew closed 2 years ago

brgew commented 2 years ago

Hi,

I am trying to install rhdf5 using R 4.2.0 on a computer running Debian Linux version 11.3. Running the install command gives an error reporting a missing symbol in the rhdf5 library.

BiocManager::install('rhdf5')

** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘rhdf5’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/me/R.400/x86_64-pc-linux-gnu-library/4.2/00LOCK-rhdf5/00new/rhdf5/libs/rhdf5.so':
  /home/me/R.400/x86_64-pc-linux-gnu-library/4.2/00LOCK-rhdf5/00new/rhdf5/libs/rhdf5.so: undefined symbol: HMAC
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/home/me/R.400/x86_64-pc-linux-gnu-library/4.2/rhdf5’

The downloaded source packages are in
        ‘/tmp/RtmpI4BNkS/downloaded_packages’
Warning message:
In install.packages(...) :
  installation of package ‘rhdf5’ had non-zero exit status

I wonder if I am missing something?

Thank you.

pbordron commented 2 years ago

I got the same error, also with an up to date debian 11.3 and R 4.2.0

R 4.2.0 is from bullseye-cran40 repository installed as described here https://cran.r-project.org/bin/linux/debian/

rhdf5 package comes from bioconductor version 3.15

Everything was alright with R 4.1.x + bioconductor 3.14.

grimbough commented 2 years ago

Do you have libopenssl installed? It looks like HMAC should be defined in that, although I'd expect that to show up as a problem when installing the Rhdf5lib before getting to rhdf5.

pbordron commented 2 years ago

It is something I checked before posting as HMAC symbol is something related to crypto, but libopenssl is installed:

$ dpkg -l libssl* openssl*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version          Architecture Description
+++-================-================-============-====================================================
ii  libssl-dev:amd64 1.1.1n-0+deb11u1 amd64        Secure Sockets Layer toolkit - development files
un  libssl-doc       <none>           <none>       (no description available)
un  libssl1.0-dev    <none>           <none>       (no description available)
un  libssl1.0.0      <none>           <none>       (no description available)
un  libssl1.0.2      <none>           <none>       (no description available)
ii  libssl1.1:amd64  1.1.1n-0+deb11u1 amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl          1.1.1n-0+deb11u1 amd64        Secure Sockets Layer toolkit - cryptographic utility
grimbough commented 2 years ago

Can you provide the full output from running BiocManager::install("rhdf5")? I want to see if the linking step when rhdf5.so is created is including libopenssl. Maybe it's installed but not being used for some reason.

pbordron commented 2 years ago

Here your are:

* installing *source* package ‘rhdf5’ ...
** using staged installation
configure: creating ./config.status
config.status: creating src/external_filters.h
** libs
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5.c -o H5.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5A.c -o H5A.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5D.c -o H5D.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5E.c -o H5E.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5F.c -o H5F.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5G.c -o H5G.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5I.c -o H5I.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5L.c -o H5L.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5O.c -o H5O.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5P.c -o H5P.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5R.c -o H5R.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5S.c -o H5S.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5S_extras.c -o H5S_extras.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5T.c -o H5T.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5Z.c -o H5Z.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c H5constants.c -o H5constants.o
g++ -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c HandleList.cpp -o HandleList.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c HandleListWrap.c -o HandleListWrap.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c bit64conversion.c -o bit64conversion.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c external_filters.c -o external_filters.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c h5dump.c -o h5dump.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c h5ls.c -o h5ls.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c h5testLock.c -o h5testLock.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c h5writeDataFrame.c -o h5writeDataFrame.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c printdatatype.c -o printdatatype.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c utils.c -o utils.o
gcc -I"/usr/share/R/include" -DNDEBUG  -I'/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/include'    -fpic  -g -O2 -ffile-prefix-map=/home/jranke/git/r-backports/bullseye/r-base-4.2.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c wrap.c -o wrap.o
g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-z,relro -o rhdf5.so H5.o H5A.o H5D.o H5E.o H5F.o H5G.o H5I.o H5L.o H5O.o H5P.o H5R.o H5S.o H5S_extras.o H5T.o H5Z.o H5constants.o HandleList.o HandleListWrap.o bit64conversion.o external_filters.o h5dump.o h5ls.o h5testLock.o h5writeDataFrame.o printdatatype.o utils.o wrap.o /home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/lib/libhdf5.a -L -lcrypto -lcurl -lsz -laec -lz -ldl -lm -L/usr/lib/R/lib -lR
installing to /home/<user>/R/x86_64-pc-linux-gnu-library/4.2/00LOCK-rhdf5/00new/rhdf5/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘rhdf5’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/00LOCK-rhdf5/00new/rhdf5/libs/rhdf5.so':
  /home/<user>/R/x86_64-pc-linux-gnu-library/4.2/00LOCK-rhdf5/00new/rhdf5/libs/rhdf5.so: undefined symbol: HMAC
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/home/<user>/R/x86_64-pc-linux-gnu-library/4.2/rhdf5’
pbordron commented 2 years ago

I solved the issue on my system.

I was using libcurl4-gnutls-dev and after replacing it with libcurl4-openssl-dev, rhdf5 successfully installs.

The strange thing is that I had no problem with previous releases.

grimbough commented 2 years ago

Thanks for the update & solution. I'm not sure which change has prompted this problem, but I'll do some testing with both curl version and adapt the configure script to try and do the correct thing each time. My intention was not to enforce a specific curl implementation, and I'd rather this was handled seamlessly.

yongchao commented 2 years ago

Thanks for the solution. It helped me with the Bioconductor packages update. I have the same experience as with grimbough that I didn't have the problem with previous releases. My OS is ubuntu 18.04. I know that I should have updated it to 20.04. Since it is working fine, I didn't bother with the OS update.

I solved the issue on my system.

I was using libcurl4-gnutls-dev and after replacing it with libcurl4-openssl-dev, rhdf5 successfully installs.

The strange thing is that I had no problem with previous releases.

grimbough commented 2 years ago

This clearly affects a few people, but I'm really struggling to recreate the scenario on my own test machine. I suspect this is really a problem with the installation of Rhdf5ib which is detecting something inappropriate and passing the wrong linking information to rhdf5.

However I've tried quite a few combinations of libcurl4-gnutls-dev, libcurl4-openssl-dev, libssl-dev etc and haven't managed to trigger the same problem myself.

pbordron commented 2 years ago

I take a bit of time and I found the way to reproduce the error and where the problem lies. The error appends when both libcurl4-gnutls-dev and libhdf5-dev are installed on the system.

Here a Dockerfile reproducting the error:

FROM debian:11.3

ARG DEBIAN_FRONTEND=noninteractive

# We update system and install gnupg2, which is a requirement for apt-key
RUN apt-get update \
  && apt-get upgrade -y \
  && apt-get install -y gnupg2 \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

# We add R repository
RUN apt-key adv \
            --keyserver 'keyserver.ubuntu.com' \
            --recv-keys '95C0FAF38DB3CCAD0C080A7BDC78B2DDEABC47B7'
RUN echo "deb http://cloud.r-project.org/bin/linux/debian bullseye-cran40/" \
     | tee /etc/apt/sources.list.d/r-project.list

# We install R
RUN apt-get update \
  && apt-get install -y  r-base \
  && apt-get clean

# -------------------------------- #
# Testing Dependencies Block (TDB) #
# -------------------------------- #

RUN apt-get install -y libssl-dev libcurl4-gnutls-dev

# Installing `libhdf5-dev` makes `rhdf5` installation fails with the HMAC undefined symbol error
RUN apt-get install -y libhdf5-dev

# Installing `libcurl4-openssl-dev` in place of `libcurl4-gnutls-dev` solves the HMAC undefined symbol error
# RUN apt-get install -y libcurl4-openssl-dev

# --------------------------------------- #
# End of Testing Dependencies Block (TDB) #
# --------------------------------------- #

RUN Rscript -e 'install.packages("BiocManager"); BiocManager::install("rhdf5", version="3.15")'

I did some quick tests in the Testing Dependencies Block (TDB) in the Dockerfile above:

Build success if:

Build fails:

grimbough commented 2 years ago

Thanks, that's an amazing help. I managed to produce the SZ_encoder_enabled undefined symbol error myself yesterday and thought it might have a similar root cause, but since it wasn't the error I was chasing I haven't dug further yet.

I'll run through a few scenarios and check what the configure script thinks it's finding.

grimbough commented 2 years ago

I think I've tracked this down. Would you mind testing https://github.com/grimbough/Rhdf5lib/tree/rhdf5-issue-109 ?

I did that with the following lines at the end of the Dockerfile and it seems to work for me.

RUN Rscript -e 'install.packages(c("BiocManager", "remotes"));'
RUN Rscript -e 'BiocManager::install("grimbough/Rhdf5lib", ref = "rhdf5-issue-109");'
RUN Rscript -e 'BiocManager::install("rhdf5", version="3.15")'
pbordron commented 2 years ago

The installation succeed now on my problematic system with libcurl4-gnutls-dev.

First, I have reinstalled the libcurl4-gnutls-dev instead libcurl4-openssl-devand reinstalled rhdf5 (i.e. BiocManager::install("rhdf5", version="3.15", force = TRUE). I get the HMAC undefined symbol error, which validate that I am in right state to test the correction.

Then, I install the patched version of Rhdf5lib (i.e. BiocManager::install("grimbough/Rhdf5lib", ref = "rhdf5-issue-109")) and reinstalled again rhdf5. The installation succeed.

I did some test with libcurl4-openssl-dev that were also successfull.

I didn't test removing libssl-dev as it will break many thing on my system, but I think the symbol problem is solved

grimbough commented 2 years ago

Awesome, thanks for taking the time to test this.

The problem stemmed from the lone -L on the line:

g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-z,relro -o rhdf5.so H5.o H5A.o H5D.o H5E.o H5F.o H5G.o H5I.o H5L.o H5O.o H5P.o H5R.o H5S.o H5S_extras.o H5T.o H5Z.o H5constants.o HandleList.o HandleListWrap.o bit64conversion.o external_filters.o h5dump.o h5ls.o h5testLock.o h5writeDataFrame.o printdatatype.o utils.o wrap.o /home/<user>/R/x86_64-pc-linux-gnu-library/4.2/Rhdf5lib/lib/libhdf5.a -L -lcrypto -lcurl -lsz -laec -lz -ldl -lm -L/usr/lib/R/lib -lR

This would appear in the scenario where you have libsz installed on your system before Rhdf5lib is installed. This was the case if you'd installed the system level libhdf5-dev. If that system library wasn't present Rhdf5lib builds & installs it's own version, and you'd get -L/path/to/Rhdf5lib/lib included in the linking step.

It's not entirely clear to me why installing libcurl4-opensl-dev fixed the issue, or why this bug wasn't producing the same issue on my Linux Mint machine. However fixing that bug seems to have corrected the linking, so I'm happy.

Thanks a lot for taking the time to help. I'll patch both the release and devel versions on Bioconductor, and hopefully this won't be an issue any more.