libgsl.so.27 not found for GSL local installation

bklempay commented 10 months ago

I had a bit of trouble installing RcppZiggurat as GSL is installed locally in my home directory, since I don't have root access. However, the installation eventually appeared to be successful by adding -I/path/to/gsl/include to the Makevars file. But when I try to load the library in R, I get the error:

Error: package or namespace load failed for ‘RcppZiggurat’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/RcppZiggurat/libs/RcppZiggurat.so':
  libgsl.so.27: cannot open shared object file: No such file or directory

I have verified that libgsl.so.27 exists in /home/bklempay/tools/gsl/lib and I have added that path to LD_LIBRARY_PATH in R:

> Sys.getenv("LD_LIBRARY_PATH")
[1] "/home/bklempay/tools/gsl/lib:/usr/lib64/R/lib:/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/lib/jvm/jre/lib/server:/usr/java/packages/lib/amd64:/lib:/usr/lib"

eddelbuettel commented 10 months ago

Edit: I misread your ticket. Your initial installation failed at the link step as R does not 'see' libgsl. See the next answer.

That is a problem with how you installed RcppZiggurart because the binary was 'naked' and did not have the required dependency declared! It a) cannot happen when you build from source and b) does not happen with 'proper' binaries as eg in my r2u where we have a proper dependency

edd@rob:~$ docker run --rm -ti rocker/r2u 
root@dbf856cbf0ac:/# apt update -qq
30 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@dbf856cbf0ac:/# apt-cache show r-cran-rcppziggurat | grep libgsl
Depends: libc6 (>= 2.29), libgcc-s1 (>= 3.0), libgsl27 (>= 2.7.1), libstdc++6 (>= 11), r-base-core (>= 4.2.0), r-api-4.0, r-cran-rcpp, r-cran-rcppgsl
root@dbf856cbf0ac:/#

so that we have (easy binary install thanks to bspm use)

root@dbf856cbf0ac:/# Rscript -e 'install.packages("RcppZiggurat")'
Ign https://r2u.stat.illinois.edu/ubuntu jammy InRelease                                                                                                                                                          
Hit https://r2u.stat.illinois.edu/ubuntu jammy Release                                                                                                                                                            
Hit http://security.ubuntu.com/ubuntu jammy-security InRelease                                                                                                                                                    
Hit http://archive.ubuntu.com/ubuntu jammy InRelease                                                                                                                                                              
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]                                                                                                                                           
Hit https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease                                                                                                                                          
Hit http://archive.ubuntu.com/ubuntu jammy-backports InRelease                                                                                                                                                    
Fetched 119 kB in 0s (0 B/s)                                                                                                                                                                                      
Install system packages as root...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Reading package lists... Done    
Building dependency tree... Done 
Reading state information... Done
Get:1 https://r2u.stat.illinois.edu/ubuntu jammy/main amd64 r-cran-rcpp amd64 1.0.11-1.ca2204.1 [1946 kB]                                                                                                         
Get:2 https://r2u.stat.illinois.edu/ubuntu jammy/main amd64 r-cran-rcppgsl amd64 0.3.13-1.ca2204.1 [199 kB]                                                                                                       
Get:3 https://r2u.stat.illinois.edu/ubuntu jammy/main amd64 r-cran-rcppziggurat amd64 0.1.6-1.ca2204.1 [410 kB]                                                                                                   
Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgslcblas0 amd64 2.7.1+dfsg-3 [94.4 kB]                                                                                                             
Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgsl27 amd64 2.7.1+dfsg-3 [1000 kB]                                                                                                                 
Fetched 3650 kB in 0s (0 B/s)                                                                                                                                                                                     
Selecting previously unselected package libgslcblas0:amd64.
(Reading database ... 18320 files and directories currently installed.)
Preparing to unpack .../libgslcblas0_2.7.1+dfsg-3_amd64.deb ...
Unpacking libgslcblas0:amd64 (2.7.1+dfsg-3) ...
Selecting previously unselected package libgsl27:amd64.
Preparing to unpack .../libgsl27_2.7.1+dfsg-3_amd64.deb ...
Unpacking libgsl27:amd64 (2.7.1+dfsg-3) ...
Selecting previously unselected package r-cran-rcpp.
Preparing to unpack .../r-cran-rcpp_1.0.11-1.ca2204.1_amd64.deb ...
Unpacking r-cran-rcpp (1.0.11-1.ca2204.1) ...
Selecting previously unselected package r-cran-rcppgsl.
Preparing to unpack .../r-cran-rcppgsl_0.3.13-1.ca2204.1_amd64.deb ...
Unpacking r-cran-rcppgsl (0.3.13-1.ca2204.1) ...
Selecting previously unselected package r-cran-rcppziggurat.
Preparing to unpack .../r-cran-rcppziggurat_0.1.6-1.ca2204.1_amd64.deb ...
Unpacking r-cran-rcppziggurat (0.1.6-1.ca2204.1) ...
Setting up r-cran-rcpp (1.0.11-1.ca2204.1) ...
Setting up libgslcblas0:amd64 (2.7.1+dfsg-3) ...
Setting up libgsl27:amd64 (2.7.1+dfsg-3) ...
Setting up r-cran-rcppgsl (0.3.13-1.ca2204.1) ...
Setting up r-cran-rcppziggurat (0.1.6-1.ca2204.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
root@dbf856cbf0ac:/#

after which the package does of course load:


root@dbf856cbf0ac:/# R

R version 4.3.2 (2023-10-31) -- "Eye Holes"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(RcppZiggurat)
> zrnorm(2)
[1] -1.1409054  0.8759027
>

This is also, I think, a duplicate issue. So I am closing -- no defect in the package or the sources but something you may want to bring up with whereever you got the binary from.

eddelbuettel commented 10 months ago

If I misread and you built from source then you need to set LD_LIBRARY_PATH to the libgsl shared library used in linking. (That is also a standard problem that arises when you 'need' system libraries but are not root, ie it is a bit of local issue. Been there done that back in the day.)

The r2u instance uses a globally available libgsl27:

root@dbf856cbf0ac:/# ldd /usr/lib/R/site-library/RcppZiggurat/libs/RcppZiggurat.so | grep libgsl
        libgsl.so.27 => /lib/x86_64-linux-gnu/libgsl.so.27 (0x00007f88bbab7000)
        libgslcblas.so.0 => /lib/x86_64-linux-gnu/libgslcblas.so.0 (0x00007f88bb0b4000)
root@dbf856cbf0ac:/#

You need to point to your local libgsl.

eddelbuettel commented 10 months ago

One such tip: https://stackoverflow.com/questions/8944271/install-library-in-home-directory

bklempay commented 10 months ago

If I misread and you built from source then you need to set LD_LIBRARY_PATH to the libgsl shared library used in linking.

Thank you for your response! I think I have done both of these things, but I still get the same error. I ran export LD_LIBRARY_PATH=/path/to/gsl/lib and then recompiled both RcppGSL and RcppZiggurat from source. This is a slight improvement. Now I get:

[bklempay@fram ~]$ ldd R/x86_64-redhat-linux-gnu-library/4.3/RcppZiggurat/libs/RcppZiggurat.so | grep libgsl
        libgsl.so.27 => /home/bklempay/tools/gsl/lib/libgsl.so.27 (0x00007f2f0fd03000)
        libgslcblas.so.0 => /home/bklempay/tools/gsl/lib/libgslcblas.so.0 (0x00007f2f0fac4000)
[bklempay@fram ~]$ ldd R/x86_64-redhat-linux-gnu-library/4.3/RcppGSL/libs/RcppGSL.so | grep libgsl
        libgsl.so.27 => /home/bklempay/tools/gsl/lib/libgsl.so.27 (0x00007fd4888fd000)
        libgslcblas.so.0 => /home/bklempay/tools/gsl/lib/libgslcblas.so.0 (0x00007fd4886be000)

I have also added LD_LIBRARY_PATH=/home/bklempay/tools/gsl/lib:${LD_LIBRARY_PATH} to my .Renviron file so that the path to the shared libraries is available every time I start up R. I confirmed this using Sys.getenv() as show in the original ticket. However, I still get the same error whenever I try to load RcppGSL or RcppZigurrat.

eddelbuettel commented 10 months ago

I can help you here as I had similar situations years ago before personal Linux machines were widespread, or in labs.

What I would do no (being a first principles guy) is to try compile and link a GSL four or five liner. I can send you examples I have from 25+ years with the GSL if you need them.

Next, you may also have a problem depending on how RcppGSL is installed. I also needs the GSL and if I recall other packages like RcppZiggurat query it. So ... can you actually load RcppGSL?

eddelbuettel commented 10 months ago

Hm, both building is good. Now I am a little puzzled. Let me think about that for a moment (and I have a call coming up anyway).

bklempay commented 10 months ago

No I get an identical error when loading RcppGSL. Similar issue... the installation seemed to work fine (after some trouble shooting). The configure executable uses gsl-config --libs to point to the shared libraries, which works fine. But when I try to load it in R, I get:

> library(RcppGSL)
Error: package or namespace load failed for ‘RcppGSL’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/RcppGSL/libs/RcppGSL.so':
  libgsl.so.27: cannot open shared object file: No such file or directory

eddelbuettel commented 10 months ago

You don't by chance have pkg-config and a different libgsl get in the way? Can you show me what src/Makevars is created / what the compile / link lines in the build are?

Also, please quote complete build instructions. You are not telling me whether the load step fails at the end of the build (when it tries to load) or afterwards. Also, it makes little sense that we think RcppGSL built, but cannot be loaded -- RcppZiggurat should then fail to build. A few things aren;t right here. I appreciate that you try to be concise but more information may help.

bklempay commented 10 months ago

Thanks for your help. I'm probably overlooking sharing some relevant details because I'm in over my head! The load step fails (with the error above) after the build is complete (seemingly successfully).

You don't by chance have pkg-config and a different libgsl get in the way?

I'm not sure how to check for this.

Here's the src/Marevars file created by ./configure:

[bklempay@fram rcppgsl]$ cat src/Makevars

# set by configure
GSL_CFLAGS = -I/home/bklempay/tools/gsl/include
GSL_LIBS   = -L/home/bklempay/tools/gsl/lib -lgsl -lgslcblas -lm

# combine with standard arguments for R
PKG_CPPFLAGS = $(GSL_CFLAGS) -I../inst/include
PKG_LIBS = $(GSL_LIBS)

... / what the compile / link lines in the build are?

Sorry I'm not sure what these are.

Here's the output when I compile both RcppGSL and RcppZiggurat:

[bklempay@fram rcppgsl]$ R CMD INSTALL .
* installing to library ‘/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3’
* installing *source* package ‘RcppGSL’ ...
** using staged installation
** libs
using C compiler: ‘gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
using C++ compiler: ‘g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
make: Nothing to be done for 'all'.
installing to /home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-rcppgsl/00new/RcppGSL/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package ‘RcppGSL’
    finding HTML links ... done
    LdFlags                                 html
    RcppGSL-package                         html
    fastLm                                  html
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RcppGSL)
[bklempay@fram rcppgsl]$ cd ../rcppziggurat/
[bklempay@fram rcppziggurat]$ R CMD INSTALL .
* installing to library ‘/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3’
* installing *source* package ‘RcppZiggurat’ ...
** using staged installation
** libs
using C++ compiler: ‘g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
make: Nothing to be done for 'all'.
installing to /home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-rcppziggurat/00new/RcppZiggurat/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package ‘RcppZiggurat’
    finding HTML links ... done
    RcppZiggurat-package                    html
    ziggurat                                html
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RcppZiggurat)

eddelbuettel commented 10 months ago

(Still on call. Please run ./cleanup or manually remove src/*.o src/*.so.)

(The src/Makevars looks good! Hopeful ...)

bklempay commented 10 months ago

Recompiled after running ./cleanup. Here's the new, slightly longer output:

[bklempay@fram rcppgsl]$ R CMD INSTALL .
* installing to library ‘/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3’
* installing *source* package ‘RcppGSL’ ...
** using staged installation
checking for gsl-config... /home/bklempay/tools/gsl/bin/gsl-config
configure: creating ./config.status
config.status: creating src/Makevars
** libs
using C compiler: ‘gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
using C++ compiler: ‘g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)’
g++ -std=gnu++17 -I"/usr/include/R" -DNDEBUG -I/home/bklempay/tools/gsl/include -I../inst/include -I'/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/Rcpp/include' -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++17 -I"/usr/include/R" -DNDEBUG -I/home/bklempay/tools/gsl/include -I../inst/include -I'/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/Rcpp/include' -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c fastLm.cpp -o fastLm.o
gcc -I"/usr/include/R" -DNDEBUG -I/home/bklempay/tools/gsl/include -I../inst/include -I'/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/Rcpp/include' -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c init.c -o init.o
g++ -std=gnu++17 -I"/usr/include/R" -DNDEBUG -I/home/bklempay/tools/gsl/include -I../inst/include -I'/home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/Rcpp/include' -I/usr/local/include    -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c setErrorHandler.cpp -o setErrorHandler.o
g++ -std=gnu++17 -shared -L/usr/lib64/R/lib -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o RcppGSL.so RcppExports.o fastLm.o init.o setErrorHandler.o -L/home/bklempay/tools/gsl/lib -lgsl -lgslcblas -lm -L/usr/lib64/R/lib -lR
installing to /home/bklempay/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-rcppgsl/00new/RcppGSL/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package ‘RcppGSL’
    finding HTML links ... done
    LdFlags                                 html
    RcppGSL-package                         html
    fastLm                                  html
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RcppGSL)

Same error when I run library(RcppGSL) in R

eddelbuettel commented 10 months ago

That is a tough nut. It

does the right thing on compile (via -I... to the right place)
does the right thing on link (via -L.... to right place)
manages to load the package just built.

So if this works, but loading in a general R session does it must mean that you LD_LIBRARY_PATH gets set correctly for the build ("fresh shell") spawned from R but NOT for the R session itself. And as I write that i realize ~/.Renviron is 'too late'. We need to set it somewhere so that is exists before R is started. E.g. ~/.profile or ~/.bashrc or ...

bklempay commented 10 months ago

That's very helpful to know, and now I understand my issue! I added this line to .bashrc, and when I try to run library(RccpGSL) directly from command line (or an interactive R session) all is good!

[bklempay@fram ~]$ tail -n 1 .bashrc
export LD_LIBRARY_PATH=/home/bklempay/tools/gsl/lib:$LD_LIBRARY_PATH
[bklempay@fram ~]$ Rscript -e 'Sys.getenv("LD_LIBRARY_PATH")'
[1] "/usr/lib64/R/lib:/usr/lib/jvm/jre/lib/amd64/server:/usr/lib/jvm/jre/lib/amd64:/usr/lib/jvm/java/lib/amd64:/usr/lib/jvm/jre/lib/server:/usr/java/packages/lib/amd64:/lib:/usr/lib:/home/bklempay/tools/gsl/lib:"
[bklempay@fram ~]$ Rscript -e 'library(RcppGSL)'
[bklempay@fram ~]$ Rscript -e 'library(RcppZiggurat)'
[bklempay@fram ~]$

Unforunately, I was foiled by the fact that I generally use RStudio Server which has its own root-level startup files that obviously do not execute ~/.bashrc though it does execute ~/.Renviron. So I'm still a little puzzled (and can't use RcppGSL or RcppZiggurat very easily without RStudio Server), but this might be a question for my system administrators... unless you have any ideas?

Thank you so much!

eddelbuettel commented 10 months ago

Yay.

As for RStudio Server: yeah, it is system-wide settings, but it will also respect user-level settings. You may have to grok the docs about which files it reads (and the whole distinction between login and non-login shells is maddening -- I 'flattened' my bash rc files years ago so that it would always source the way (or so I hope)).

cldebban commented 1 month ago

I just had the same problem with installing RcppZiggurat in RStudio on my university's high-performance computing cluster. After reading this thread, I decided to access R on the cluster through the terminal instead to install the package, and that seems to have worked. Now that I installed it that way, it also works on RStudio. I'm not familiar enough with how exactly these systems are set up to understand why that worked, but it might help others with the same issue in the future!

eddelbuettel commented 1 month ago

That is a common problem when the application package manager (say install.packages(), ie the R mechanism) is disconnected from the system package manager.

Say you install RcppGSL whem GNU GSL 2.7.* is installed system wide. RcppGSL will use it, it will have a linker dependency on libsgl27 and eg ldd shows it.

Then the system administrator updates the and GNU GSL rolls from 2.7. to 2.8.0 because _the system level package manager does not know 2.7. is still used by R package RcppGSL_. Hence the breakage.

You cannot avoid this unless application and system package managers are integrated. Which, coincidentally, my r2u system does. With a (compiled) r-cran-rcppgsl package its dependency on, say, libgsl27 would be known, and the library would not be removed.

So your HPC Administrator is between a rock and a hard place. Either he makes you happy by getting you current software yet hurts you with potential breakage as describe. Or he keeps the system 'constant' but it is getting stale. A hard problem.

eddelbuettel / rcppziggurat

libgsl.so.27 not found for GSL local installation #20