math detection ramblings

[loriab]

Hi, @bast, @robertodr asked me to collect some of my thoughts on renovated math detection, esp. in the light of new CMake features. So here's some ramblings, and I'll post to this issue as other things occur to me.

• Really must form CMake targets of blas/lapack (I don't really care if they're together or separate). Reasons:
  • There's a lot of potentially user-set vars to guide math detection. In the case of a cmake superbuild (or even ordinary passing the beneficence of autocmake math detection to another cmake build who doesn't want all that in their repo), don't want to haul all those variables into every package. Instead, want to detect once, then form a math target that can be passed with only a e.g., TargetLAPACK_DIR var.
  • lists of strings like LAPACK_LIBRARIES=-Wl,--start-list;-lmkl_lapack_95;-Wl,--end-list;-Wl,--start-list;-lmkl_thread;-liomp5;-Wl,--end-list get "optimized" by cmake. It considers each of those items a library, thinks it knows the rules of rearranging libraries, and helps out by optimizing away your link directives. Very bad. CMake respects targets much more than strings.
• Seek to detangle OMP/BLAS
  • New FindOpenMP in cmake 3.8 or 10 can help. In particular, for clang, can (by setting a variable before detection) influence whether libomp or libiomp5 is added to OMP target.
  • In psi's case, want to make sure that can build psi with Intel compilers but write out a Math CMake target such that a plugin built with a compatible compiler (e.g., gcc on linux, clang on Mac) also has the right omp flags (and doesn't get two competing omp libs linked). This has led to the below, which is mainly a transcription of the MKL link link advisor (c. summer 2017) plus some added comments about extra flags needed out to the right. I haven't fully implemented this anywhere, just was figuring out the scope of the problem.

+# [Linux|Mac]/compiler_fam/[dynamic|single_dynamic_lib]/[native|no_choice|intel]_OpenMP_threading
+
+# (a) = iomp5 only choice with icpc and that's self-contained but want the extra flags so g++-based plugins run correctly
+# (b) = clang-based so no intervention req'd
+# (c) = to satisfy OpenMP symbols and suppress libgomp
+ 
+# L/gnu/Dyn/N       -lmkl_intel_lp64 -lmkl_gnu_thread   -lmkl_core -lgomp         -lpthread -lm -ldl
+# L/pgi/Dyn/N       -lmkl_intel_lp64 -lmkl_pgi_thread   -lmkl_core -pgf90libs -mp -lpthread -lm -ldl
+ 
+# L/intel/Dyn/-     -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       IN
+# M/intel/Dyn/-     -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       IN
+# L/gnu/Dyn/I       -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       I(c):[-fno-openmp] N(above)
+# M/gnu/Dyn/-       -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       I(c):[-fno-openmp] N(n/a):
+# L/pgi/Dyn/I       -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       I:? N(above)
+# M/pgi/Dyn/-       -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       I:? N(n/a):
+# M/clang/Dyn/-     -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl       IN(b):[]
+ 
+# L/intel/SDL/-     -lmkl_rt -lpthread -lm -ldl         IN(a):[-liomp5 -fno-openmp]
+# M/intel/SDL/-     -lmkl_rt -lpthread -lm -ldl         IN(b):[]
+# L/gnu/SDL/-       -lmkl_rt -lpthread -lm -ldl         I(c):[-liomp5 -fno-openmp] N:[]
+# M/gnu/SDL/-       -lmkl_rt -lpthread -lm -ldl         I(c):[-liomp5 -fno-openmp] N(n/a):
+# L/pgi/SDL/-       -lmkl_rt -lpthread -lm -ldl         ?
+# M/pgi/SDL/-       -lmkl_rt -lpthread -lm -ldl         ?
+# M/clang/SDL/-     -lmkl_rt -lpthread -lm -ldl         IN(b):[]

• May want to consider some extra guidance vars

  #    - LAPACK_LIBRARIES "Location of BLAS/LAPACK libraries as ";"-separated list of full paths, bypassing math detection"
  #    - LAPACK_INCLUDE_DIRS "Location of BLAS/LAPACK headers (only needed for MKL), bypassing math detection"
  +#    - LAPACK_PREFER_SINGLE_DYNAMIC_LIBRARY_MKL "Seek the mkl_rt library over dynamic (default: on)"
  +#    - LAPACK_PREFER_STATIC_MKL "Seek static MKL libraries rather than dynamic or single dynamic (default: off)"
  +#    - LAPACK_PREFER_NATIVE_OMP_WITH_MKL "Use native OpenMP library (e.g., gomp) not iomp5 w/MKL if allowed (default: off)"

[bast]

Thanks a lot Lori! That module is up for a serious redesign very soon (before xmas) and your notes and recommendations are super useful for this. If you have more thoughts, please append. They will all be considered.

[robertodr]

I think this could be "escalated" to a CMake issue/merge request. The default BLAS/LAPACK module is simply awful.

[loriab]

Second the "simply awful" appraisal. But I can't imaging supporting Windows, ilp64, all combinations of static, cdft, etc. to replace the CMake one.

Slightly off-topic for autocmake, but the fall-back approach seems to work for modules who don't want to check in the enhanced math detection, https://github.com/ilyak/libefp/blob/master/cmake/FindTargetLAPACK.cmake

[miroi]

Yes, old mathlibs detection is utilized in my mini project, https://github.com/miroi/mathlibs-tester ...

I have no comprehensive idea as how to make universal cmake mathlibs detection macro ....

[bast]

My goal is less entanglement, higher cohesion, and give more control to the caller and leave less thinking to this library. All suggestions welcome which help to brainstorm this better.

[loriab]

fyi https://gist.github.com/multiplemonomials/6ee074038778a21cd25b73659a6a82d9

[bast]

Thanks!

[loriab]

a thought on matching math detection to a pre-built numpy. preserving the comment here. https://github.com/psi4/psi4/pull/1031#discussion_r192517976