using MKL with rocker/r-devel

[merliseclyde]

I was tying to use the (updated) script to add MLK support in a rocker/r-devel image (so far just running interactively) to test an R package (/tthps://github.com/merliseclyde/bark) that fails unit-tests on CRAN MKL. (I am guessing it is a random-number threading issue, but would like to debug more)

Script (slightly updated for MKL version) from the one here

# add MKL

apt-get update && apt-get install -y gnupg

cd /tmp
# now get the key:
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
# now install that key
apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
# now remove the public key file exit the root shell
# rm GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB

# wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list
## just MKL
sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'

apt-get update

# use apt-get install intel-mkl to see most recent targets :-)
apt-get install intel-mkl-2020.4-912 2020.4-912

## update alternatives
update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so     \
                    libblas.so-x86_64-linux-gnu      /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so.3   \
                    libblas.so.3-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so   \
                    liblapack.so-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so.3 \
                    liblapack.so.3-x86_64-linux-gnu  /opt/intel/mkl/lib/intel64/libmkl_rt.so 50

# Next, we have to tell the dyanmic linker about two directories use by the MKL, and have it update its cache:

echo "/opt/intel/lib/intel64"     >  /etc/ld.so.conf.d/mkl.conf
echo "/opt/intel/mkl/lib/intel64" >> /etc/ld.so.conf.d/mkl.conf
ldconfig

But when I go to start r-devel. via RD and run sesssionInfo() I am not seeing the MKL BLAS but: Matrix products: default BLAS: /usr/local/lib/R/lib/libRblas.so LAPACK: /usr/local/lib/R/lib/libRlapack.so; LAPACK version 3.11.0 Any suggestions of what I may have missed? Is there an existing rocker/r version with MKL support already configured ?

TIA - Merlise

[eddelbuettel]

Hi Merlise -- you are the second person who is coming ringing this week.

I clearly need to hightlight better what is at the botton of the README under appendix: this is essentially five years old, and outdated. You should be able to get MKL directly from the distro. My 22.10 system on which I type this has

edd@rob:~$ apt-cache search intel-mkl
intel-mkl - Intel® Math Kernel Library (Intel® MKL)
intel-mkl-cluster - Intel® Math Kernel Library (Intel® MKL) (Cluster)
intel-mkl-doc - Intel® Math Kernel Library (Intel® MKL) (Doc)
intel-mkl-full - Intel® Math Kernel Library (Intel® MKL) (Full)
libmkl-cluster-dev - Static libs of intel-MKL: Cluster
libmkl-computational-dev - Static libs of intel-MKL: Computational Layer
libmkl-interface-dev - Static libs of intel-MKL: Interface Layer
libmkl-meta-cluster - Metapackage for Intel-MKL: Shared Object for Cluster
libmkl-meta-computational - Metapackage of intel-MKL: Shared Objects for Computaional Layer
libmkl-meta-interface - Metapackage of intel-MKL: Shared Objects of Interface Layer
libmkl-meta-threading - Metapackage of intel-MKL: Shared Objects of Threading Layer
libmkl-threading-dev - Static libs of intel-MKL: Threading Layer
edd@rob:~$ 

[eddelbuettel]

With rocker/r-devel there may be another simplification we always had in the same base shell script I posted for ten or fiften years to build R (here R-devel) from source: I think for simplicity it used the included LAPACK/BLAS and not the interface.

Again looking at my local installation -- this confirms it. But I suggest you check your container:

> sessionInfo()
R Under development (unstable) (2023-04-09 r84207)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.10

Matrix products: default
BLAS:   /usr/local/lib/R-devel/lib/R/lib/libRblas.so 
LAPACK: /usr/local/lib/R-devel/lib/R/lib/libRlapack.so;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Chicago
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.4.0 parallel_4.4.0 fortunes_1.5-4
> 

[eddelbuettel]

Pulling the newest (remotely cron built) r-devel now.

... Moments later ... conforming. Also libRblas.so and libRlapack.so so no 'hook' for MKL or OpenBLAS.

edd@rob:~$ dkrrr rocker/r-devel RD

R Under development (unstable) (2023-04-09 r84210) -- "Unsuffered Consequences"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> sessionInfo()
R Under development (unstable) (2023-04-09 r84210)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux bookworm/sid

Matrix products: default
BLAS:   /usr/local/lib/R/lib/libRblas.so 
LAPACK: /usr/local/lib/R/lib/libRlapack.so;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.4.0
> 

[eddelbuettel]

I made a https://github.com/eddelbuettel/mkl4deb/commit/9178efd1a37957fb400ca414a35b2287f3e8e14e to clarify in the README.

So I think we can close this. Now, as for bark, CRAN actually does not use MKL as far as I know [ goes checking ]. Oh, wow. One of those added BDR tests. Well in that case I recommend to (hushes quietly) ... simply not run that test at CRAN. I have a trick in Rcpp to turn extended tests on when the version number has four parts (say: 1.0.10.4 as our current dev version) and off when it has three (say: 1.0.11, next release due July). That avoids control over env vars and other tricks. Or you could move that one test to another directory and use a helper script at home to include it etc pp.

Ok if we close this? Feel free to reopen if you have an issue with the old scripts, happy to help at Rocker repo or otherwise.