building FlexiBLAS 3.0.4 with -DMKL_CUSTOM=ON fails

[boegel]

I'm seeing this when trying to build FlexiBLAS 3.0.4 on top of (only) imkl 2020.4.304 with GCC 10.2:

[ 73%] Linking C executable benchmark.INTEL_MKL_SEQ
cd /tmp/easybuild_obj/examples && /software/CMake/3.18.4-GCCcore-10.2.0/bin/cmake -E cmake_link_script CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/link.txt --verbose=1
/software/GCCcore/10.2.0/bin/gcc -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=c99 -D_FILE_OFFSET_BITS=64 -fopenmp -Wno-unused-parameter -O3 -DNDEBUG -L/software/imkl/2020.4.304-gompi-2020b/lib64 -L/software/imkl/2020.4.304-gompi-2020b/lib -L/software/BLIS/0.8.0-GCCcore-10.2.0/lib64 -L/software/BLIS/0.8.0-GCCcore-10.2.0/lib -L/software/OpenBLAS/0.3.12-GCC-10.2.0/lib64 -L/software/OpenBLAS/0.3.12-GCC-10.2.0/lib -L/software/GCCcore/10.2.0/lib64 -L/software/GCCcore/10.2.0/lib -rdynamic CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o -o benchmark.INTEL_MKL_SEQ  ../libcscutils/lib/libcscutils.a -lm -ldl -lgfortran -lm -lpthread -lgfortran -lm -lquadmath -lm
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemv: error: undefined reference to 'dgemv_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemv: error: undefined reference to 'dgemv_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemv: error: undefined reference to 'dgemv_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemv: error: undefined reference to 'dgemv_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemm: error: undefined reference to 'dgemm_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemm: error: undefined reference to 'dgemm_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemm: error: undefined reference to 'dgemm_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_dgemm: error: undefined reference to 'dgemm_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_daxpy: error: undefined reference to 'daxpy_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_daxpy: error: undefined reference to 'daxpy_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_daxpy: error: undefined reference to 'daxpy_'
CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/benchmark.c.o:benchmark.c:function benchmark_daxpy: error: undefined reference to 'daxpy_'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/build.make:107: examples/benchmark.INTEL_MKL_SEQ] Error 1
make[2]: Leaving directory '/tmp/easybuild_obj'
make[1]: *** [CMakeFiles/Makefile2:1044: examples/CMakeFiles/benchmark.INTEL_MKL_SEQ.dir/all] Error 2
make[1]: Leaving directory '/tmp/easybuild_obj'

Complete cmake command to configure the build was:

cmake -DCMAKE_INSTALL_PREFIX=/software/FlexiBLAS/3.0.4-gompi-2020b -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE -DBLAS_AUTO_DETECT=ON -DMKL_CUSTOM=ON /tmp/flexiblas-3.0.4/

$MKLROOT was set to /software/imkl/2020.4.304-gompi-2020b/mkl, so the mkl subdirectory of the imkl installation.

Am I overlooking something? Do I need to do something extra to get the MKL wrapper libraries built?

My goal is to have a FlexiBLAS build on top of several backends (OpenBLAS, BLIS, imkl), where you can easily switch between them using export FLEXIBLAS=BLIS or export FLEXIBLAS=imkl. Is using -DMKL_CUSTOM=ON the right approach for that? I also tried using -DEXTRA='BLIS;OpenBLAS,imkl' along with specifying -Dimkl_LIBRARY="...", but then I run into runtime errors like this:

INTEL MKL ERROR: /software/imkl/2020.4.304-gompi-2020b/mkl/lib/intel64/libmkl_avx2.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

[grisuthedragon]

That were two CMAKE bugs in a row. I fix them in my experimental codes. Normally the MKL_CUSTOM will not be necessary since the normal MKL detection will do the work. But this was also broken when setting the `MKLROOT´ variable. The bug will be fixed in 3.1. upcomming soon.

[boegel]

@grisuthedragon We aim to install FlexiBLAS using -DBLAS_AUTO_DETECT=OFF, so we're in control over what it's picking up on. Will -DMKL_CUSTOM=ON be all we need in combination with that when installing FlexiBLAS on top of Intel MKL + other BLAS libraries (like OpenBLAS/BLIS)?

[grisuthedragon]

@boegel The MKL_CUSTOM is not required in this case. For the plan you have, the work done by MKL_CUSTOM needs to be done by hand. We plan to include a better description of this procedure in the next release. But most details about building flexiblas without the automatic detection is documented in https://github.com/mpimd-csc/flexiblas/issues/13

I plan so provide a "post install" script for building additional BLAS backends easily.

[grisuthedragon]

In tools/backend_builder FlexiBLAS now contains a CMake toolbox for easy building of backends without rebuilding FlexiBLAS.

[grisuthedragon]

Closed with 3.1.0