use oneMKL to provide CPU LAPACK functions

[stuart-nolan]

Originally posted by @gardner48 in https://github.com/LLNL/sundials/issues/173#issuecomment-1176270242

Naively following the first suggestion in #173 by doing something like cmake ... -DBLA_VENDOR=Intel10_64ilp ... results in

CMake Warning:
  Manually-specified variables were not used by the project:

    BLA_VENDOR

FindBLAS exists on my system (ubuntu 22.04) but apparently is not implemented/used by sundials. FWIW FindBLAS does work with MKL for at least one other lib (suitesparse) on my system.

The second suggestion, specify the lapack libraries along with appropriate (global) cflags, works; however results in all built dynamic libraries having links to MKL. There does not seem to be cmake options for linking and c flags specific to lapack. One can do two builds e.g. one for all non lapack libraries, a second for only the two sundials lapack libraries, and then use the two lib sets together (this is what I've done and so far works) but seems like a kludge.

Can the sundials cmake build system better handle linking to MKL? Have I missed some cmake option or should I label this issuse as a feature request?

[balos1]

Did you set ENABLE_LAPACK to ON? BLA_VENDOR should be used. Since SUNDIALS doesn't ever use BLAS without also needing LAPACK, we only call FindLAPACK.

[stuart-nolan]

Thank you for responding. I tried multiple ways, most using the cmake command line and nearly all with -DENABLE_LAPACK=ON. That said, I also tried the ccmake curses gui with a simpler config than I give below.

For the cmake cli, I use a script to generate the cmake command, they are long but here are examples:

cmake cli examples

With -DBLA_VENDOR=Intel10_64ilp ``` /usr/bin/cmake -DBUILD_STATIC_LIBS=OFF -DENABLE_OPENMP=ON -DCMAKE_INSTALL_PREFIX:PATH=/home/ul/.local -DCMAKE_INSTALL_LIBDIR:PATH=lib/sundials-6.6.1.mkl -DCMAKE_INSTALL_INCLUDEDIR:PATH=include/sundials-6.6.1.mkl -DEXAMPLES_INSTALL_PATH:PATH=/home/ul/.local/examples/sundials-6.6.1.mkl -DENABLE_KLU=ON -DKLU_INCLUDE_DIR=/home/ul/.local/include/suitesparse-7.2.0.mkl -DKLU_LIBRARY_DIR=/home/ul/.local/lib/suitesparse-7.2.0.mkl -DENABLE_LAPACK=ON -DBLA_VENDOR=Intel10_64ilp -DCMAKE_C_FLAGS="-march=native -fPIC -O3 -DNDEBUG" ../ ``` Specifying the lapack libraries (works but is a pain to make an install that does not have all libraries linked to mkl): ``` /usr/bin/cmake -DBUILD_STATIC_LIBS=OFF -DENABLE_OPENMP=ON -DCMAKE_INSTALL_PREFIX:PATH=/home/ul/.local -DCMAKE_INSTALL_LIBDIR:PATH=lib/sundials-6.6.1.mkl -DCMAKE_INSTALL_INCLUDEDIR:PATH=include/sundials-6.6.1.mkl -DEXAMPLES_INSTALL_PATH:PATH=/home/ul/.local/examples/sundials-6.6.1.mkl -DENABLE_KLU=ON -DKLU_INCLUDE_DIR=/home/ul/.local/include/suitesparse-7.2.0.mkl -DKLU_LIBRARY_DIR=/home/ul/.local/lib/suitesparse-7.2.0.mkl -DENABLE_LAPACK=ON -DLAPACK_LIBRARIES="/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gnu_thread.so" -DCMAKE_C_FLAGS="-march=native -fPIC -O3 -DNDEBUG -DMKL_ILP64 -m64 -I/opt/intel/oneapi/mkl/2023.2.0/include -L/opt/intel/oneapi/mkl/2023.2.0/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl" ../ ```

My appologies, I just reran the cmake cli example with -DBLA_VENDOR=Intel10_64ilp now and noticed the error output:

... <truncated>
-- Found Threads: TRUE  
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gnu_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so;-lm;-ldl  
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find LAPACK (missing: LAPACK_LIBRARIES)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.22/Modules/FindLAPACK.cmake:744 (find_package_handle_standard_args)
  cmake/tpl/SundialsLapack.cmake:50 (find_package)
  cmake/SundialsSetupTPLs.cmake:122 (include)
  CMakeLists.txt:171 (include)

specifying the lapack libraries (without the mkl link advisor flags included in -DCMAKE_C_FLAGS) gives me

...
  ERROR: SUNDIALS interface to LAPACK is not functional.
...

I can always add the link advisor flags, but then there is no point to using -DBLA_VENDOR=Intel10_64ilp is there?

Let me know if you have a suggestion for me to try.

For the record, I'm using sundials 6.6.1 and oneMKL 2023.0 Update 2 Product build 20230613 installed via sudo apt install intel-oneapi-mkl intel-oneapi-mkl-devel

[gardner48]

The build system has been updated to use the CMake imported LAPACK target which should make building with MKL more robust. In particular, the BLAS_LINKER_FLAGS and LAPACK_LINKER_FLAGS variables can now be used to set linker flags specifically for BLAS and LAPACK.