MKL

[jjakowski]

Hi,

I am using RMG on a local workstation and I would love to use MKL instead of default (inefficient) libraries openblas, lapack, fftw, scalapack? How can force CMake to find MKL and use it ?

Jacek

[elbriggs]

Hi Jacek RMG can be built with MKL but it can be tricky when there are multiple versions of the libraries on the machine. As far as performance goes you won't see much benefit from the intel fftw libraries since RMG does not spend much time in ffts. Blas is a different story though and on some platforms MKL is a better choice. If the environment variable MKLROOT is set to point to the MKL installation cmake should try to pick the libraries up automatically but this has only been tested on a few platforms. What OS version are you running?

Emil

[jjakowski]

Hi Emil, I have ubuntu 18.04. Thank you for suggestion. As you suggested I have set MKLROOT:

%> export MKLROOT=/opt/intel/mkl

and then run cmake

%> cd build ; cmake ..

But I am not sure if cmake found MKL. As you can see the output does not show it:

(anaconda3)j2c@bigroo:~/Development/RMG/rmgdft-tmp/build$ cmake .. -- The Fortran compiler identification is GNU 7.4.0 -- Check for working Fortran compiler: /usr/bin/f95 -- Check for working Fortran compiler: /usr/bin/f95 -- works -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Checking whether /usr/bin/f95 supports Fortran 90 -- Checking whether /usr/bin/f95 supports Fortran 90 -- yes -- The CXX compiler identification is GNU 7.4.0 -- The C compiler identification is GNU 7.4.0 -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Found Git: /usr/bin/git (found version "2.17.1") -- Performing Test COMPILER_SUPPORTS_CXX14 -- Performing Test COMPILER_SUPPORTS_CXX14 - Success -- Performing Test COMPILER_SUPPORTS_CXX0X -- Performing Test COMPILER_SUPPORTS_CXX0X - Success -- Performing Test COMPILER_SUPPORTS_C99 -- Performing Test COMPILER_SUPPORTS_C99 - Success -- Performing Test COMPILER_SUPPORTS_DASH_C99 -- Performing Test COMPILER_SUPPORTS_DASH_C99 - Failed -- Detecting Fortran/C Interface -- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling -- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") -- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1") -- Found MPI_Fortran: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_usempif08.so (found version "3.1") -- Found MPI: TRUE (found version "3.1") -- Found OpenMP_C: -fopenmp (found version "4.5") -- Found OpenMP_CXX: -fopenmp (found version "4.5") -- Found OpenMP_Fortran: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - found -- Found Threads: TRUE -- Found NUMA: /usr/lib/x86_64-linux-gnu/libnuma.so -- Found HWLOC: /usr/lib/x86_64-linux-gnu/libhwloc.so -- Performing Test RMG_ASYNC_ALLREDUCE_INTERNAL -- Performing Test RMG_ASYNC_ALLREDUCE_INTERNAL - Success -- Boost version: 1.65.1 -- Found the following Boost libraries: -- thread -- system -- iostreams -- program_options -- filesystem -- chrono -- date_time -- atomic -- regex -- Found FFTW: /usr/lib/x86_64-linux-gnu/libfftw3.a -- Looking for Fortran sgemm -- Looking for Fortran sgemm - found -- A library with BLAS API found. -- Performing Test RMG_OPENBLAS_SET_NUM_THREADS_INTERNAL -- Performing Test RMG_OPENBLAS_SET_NUM_THREADS_INTERNAL - Success -- HDF5: Using hdf5 compiler wrapper to determine C configuration -- Found HDF5: /home/j2c/anaconda3/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/home/j2c/anaconda3/lib/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.2") found components: C -- Using HDF5 non-scalable serial I/O due to the lack of library support for parallel -- Compiling with C standard: 99 -- Compiling with C++ standard: 14 -- Found OpenMP_C: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") found components: C -- Performing Test HAVE_MATH -- Performing Test HAVE_MATH - Failed -- Performing Test HAVE_LIBM_MATH -- Performing Test HAVE_LIBM_MATH - Success -- Build type: Release -- ========= -- Compiling and Building BLACS INSTALL Testing to set correct variables -- Configure in the INSTALL directory successful -- Build in the BLACS INSTALL directory successful -- ========= -- Testing FORTRANMANGLING -- CDEFS set to Add -- ========= -- Creating tests for RMG -- Configuring done -- Generating done -- Build files have been written to: /home/j2c/Development/RMG/rmgdft-tmp/build

I also grepped for MKL or mkl but there is not indication that my build can see MKL. Is there a way to verify that cmake configures my makefile for MKL? Also, I do see that make actually builds scalapack using the source code provided with RMG -- if this is any indication that MKL is not used. Any suggestion?

Jacek

On Wed, Oct 2, 2019 at 5:59 PM Emil Briggs notifications@github.com wrote:

Hi Jacek RMG can be built with MKL but it can be tricky when there are multiple versions of the libraries on the machine. As far as performance goes you won't see much benefit from the intel fftw libraries since RMG does not spend much time in ffts. Blas is a different story though and on some platforms MKL is a better choice. If the environment variable MKLROOT is set to point to the MKL installation cmake should try to pick the libraries up automatically but this has only been tested on a few platforms. What OS version are you running?

Emil

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RMGDFT/rmgdft/issues/39?email_source=notifications&email_token=ABQW7AEBZQSGTFBPNBML233QMUKVNA5CNFSM4I43RJYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAGKM4Y#issuecomment-537699955, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQW7AAYCLIXLCKLV55F3ALQMUKVNANCNFSM4I43RJYA .

[elbriggs]

One possibility is to run ldd on the executable which should give you a listing of the dynamically linked libraries. That won't help if mkl was statically linked though. I've got a Ubuntu 18.04 VM on my workstation and I'll try downloading MKL and seeing what I find.

As far as scalapack goes that is not definitive. RMG always builds scalapack from source since we ran into so many problems when trying to use system provided libraries. (Multiple versions of scalapack on many machines with each version requiring specific MPI and blas libraries). For workstation class problems scalapack is not a performance limiter either as long as it is linked against a good set of blas libraries but we may add an option to use a system scalapack library at some point.

[elbriggs]

I managed to get rmg to link against MKL on Ubuntu 18.04 by doing the following.

source /opt/intel/bin/compilervars.sh -arch intel64 -platform linux export BLA_VENDOR=Intel10_64lp

However the resulting executable crashed almost immediately when I tried to run it. Looking into the reasons for that now.

[jjakowski]

Mkl has 2 different interfaces related to whether integer or long integers are used for addressing arrays. There is 64ilp and 32ilp. Is it possible that the crash is due to wrong interface?

-----Original Message----- From: "Emil Briggs" notifications@github.com Sent: ‎10/‎3/‎2019 8:19 AM To: "RMGDFT/rmgdft" rmgdft@noreply.github.com Cc: "jjakowski" jjakowski@gmail.com; "Author" author@noreply.github.com Subject: Re: [RMGDFT/rmgdft] MKL (#39)

I managed to get rmg to link against MKL on Ubuntu 18.04 by doing the following. source /opt/intel/bin/compilervars.sh -arch intel64 -platform linux export BLA_VENDOR=Intel10_64lp However the resulting executable crashed almost immediately when I tried to run it. Looking into the reasons for that now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[elbriggs]

I checked and that was not the issue. It turns out the problem is caused by a conflict with the fft headers. Ubuntu includes the ffftw3 headers in /usr/include but these do not work when linking against the Intel fft routines. I fixed this manually but I'll try to get a fix implemented in the code.

[elbriggs]

The latest commit includes a fix for this. Let me know if that works on your machine. I did not see much performance difference on my workstation between openblas and MKL but I have an AMD processor and perhaps MKL does not generate such ideal code in that case.

[prckent]

Aside: Can confirm that when used with the Intel compilers, MKL is successfully linked, as is usual.