search

icl-utk-edu / slate

SLATE is a distributed, GPU-accelerated, dense linear algebra library targetting current and upcoming high-performance computing (HPC) systems. It is developed as part of the U.S. Department of Energy Exascale Computing Project (ECP).
https://icl.utk.edu/slate/
BSD 3-Clause "New" or "Revised" License
84 stars 20 forks source link

Frontier: Cannot find -lscalapack #198

Open devreal opened 1 month ago

devreal commented 1 month ago

Description

Linking the tester executable fails with -lscalapack missing on Frontier:

[ 83%] Linking CXX executable tester
cd /ccs/home/jschuchart/src/slate/slate/build/test && /autofs/nccs-svm1_sw/frontier/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/cmake-3.23.2-4r4mpiba7cwdw2hlakh5i7tchi64s3qd/bin/cmake -E cmake_link_script CMakeFiles/tester.dir/link.txt --verbose=1
/opt/cray/pe/craype/2.7.31/bin/CC CMakeFiles/tester.dir/matrix_params.cc.o CMakeFiles/tester.dir/matrix_utils.cc.o CMakeFiles/tester.dir/test.cc.o CMakeFiles/tester.dir/test_add.cc.o CMakeFiles/tester.dir/test_bdsqr.cc.o CMakeFiles/tester.dir/test_copy.cc.o CMakeFiles/tester.dir/test_gbmm.cc.o CMakeFiles/tester.dir/test_gbnorm.cc.o CMakeFiles/tester.dir/test_gbsv.cc.o CMakeFiles/tester.dir/test_ge2tb.cc.o CMakeFiles/tester.dir/test_gecondest.cc.o CMakeFiles/tester.dir/test_gelqf.cc.o CMakeFiles/tester.dir/test_gels.cc.o CMakeFiles/tester.dir/test_gemm.cc.o CMakeFiles/tester.dir/test_geqrf.cc.o CMakeFiles/tester.dir/test_gesv.cc.o CMakeFiles/tester.dir/test_getri.cc.o CMakeFiles/tester.dir/test_hb2st.cc.o CMakeFiles/tester.dir/test_hbmm.cc.o CMakeFiles/tester.dir/test_hbnorm.cc.o CMakeFiles/tester.dir/test_he2hb.cc.o CMakeFiles/tester.dir/test_heev.cc.o CMakeFiles/tester.dir/test_hegst.cc.o CMakeFiles/tester.dir/test_hegv.cc.o CMakeFiles/tester.dir/test_hemm.cc.o CMakeFiles/tester.dir/test_her2k.cc.o CMakeFiles/tester.dir/test_herk.cc.o CMakeFiles/tester.dir/test_hesv.cc.o CMakeFiles/tester.dir/test_norm.cc.o CMakeFiles/tester.dir/test_pbsv.cc.o CMakeFiles/tester.dir/test_pocondest.cc.o CMakeFiles/tester.dir/test_posv.cc.o CMakeFiles/tester.dir/test_potri.cc.o CMakeFiles/tester.dir/test_scale.cc.o CMakeFiles/tester.dir/test_scale_row_col.cc.o CMakeFiles/tester.dir/test_set.cc.o CMakeFiles/tester.dir/test_stedc.cc.o CMakeFiles/tester.dir/test_stedc_deflate.cc.o CMakeFiles/tester.dir/test_stedc_secular.cc.o CMakeFiles/tester.dir/test_stedc_sort.cc.o CMakeFiles/tester.dir/test_stedc_z_vector.cc.o CMakeFiles/tester.dir/test_steqr.cc.o CMakeFiles/tester.dir/test_sterf.cc.o CMakeFiles/tester.dir/test_svd.cc.o CMakeFiles/tester.dir/test_symm.cc.o CMakeFiles/tester.dir/test_syr2k.cc.o CMakeFiles/tester.dir/test_syrk.cc.o CMakeFiles/tester.dir/test_tb2bd.cc.o CMakeFiles/tester.dir/test_tbsm.cc.o CMakeFiles/tester.dir/test_trcondest.cc.o CMakeFiles/tester.dir/test_trmm.cc.o CMakeFiles/tester.dir/test_trsm.cc.o CMakeFiles/tester.dir/test_trtri.cc.o CMakeFiles/tester.dir/test_unmqr.cc.o CMakeFiles/tester.dir/test_unmtr_hb2st.cc.o CMakeFiles/tester.dir/test_unmtr_he2hb.cc.o CMakeFiles/tester.dir/pclange.f.o CMakeFiles/tester.dir/pclansy.f.o CMakeFiles/tester.dir/pclantr.f.o CMakeFiles/tester.dir/pdlange.f.o CMakeFiles/tester.dir/pdlansy.f.o CMakeFiles/tester.dir/pdlantr.f.o CMakeFiles/tester.dir/pslange.f.o CMakeFiles/tester.dir/pslansy.f.o CMakeFiles/tester.dir/pslantr.f.o CMakeFiles/tester.dir/pzlange.f.o CMakeFiles/tester.dir/pzlansy.f.o CMakeFiles/tester.dir/pzlantr.f.o -o tester  -Wl,-rpath,/ccs/home/jschuchart/src/slate/slate/build/testsweeper:/ccs/home/jschuchart/src/slate/slate/build:/ccs/home/jschuchart/src/slate/slate/build/lapackpp:/ccs/home/jschuchart/src/slate/slate/build/blaspp ../testsweeper/libtestsweeper.so.1.0.0 ../libslate_matgen.so.1.0.0 -lscalapack ../libslate.so.1.0.0 ../lapackpp/liblapackpp.so.1.0.0 ../blaspp/libblaspp.so.1.0.0 /opt/cray/pe/libsci/23.12.5/CRAY/17.0/x86_64/lib/libsci_cray_mpi_mp.so /opt/cray/pe/libsci/23.12.5/CRAY/17.0/x86_64/lib/libsci_cray_mp.so /opt/cray/pe/cce/17.0.0/cce/x86_64/lib/libcraymp.so /opt/rocm-6.0.0/lib/librocblas.so.4.0.60000 /opt/rocm-6.0.0/lib/libamdhip64.so.6.0.60000 /opt/rocm-6.0.0/lib/librocsolver.so.0.1.60000 /opt/rocm-6.0.0/lib/librocblas.so.4.0.60000 /opt/cray/pe/cce/17.0.0/cce-clang/x86_64/lib/clang/17/lib/linux/libclang_rt.builtins-x86_64.a /opt/rocm-6.0.0/lib/libamdhip64.so.6.0.60000 
ld.lld: error: unable to find library -lscalapack
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Steps To Reproduce

On Frontier with the following module list:

$ module list

Currently Loaded Modules:
  1) craype-x86-trento    4) perftools-base/23.12.0                  7) cce/17.0.0        10) cray-mpich/8.1.28    13) Core/24.00                 16) hsi/default        19) cmake/3.23.2
  2) libfabric/1.15.2.0   5) xpmem/2.6.2-2.5_2.40__gd067c3f.shasta   8) craype/2.7.31     11) cray-libsci/23.12.5  14) tmux/3.2a                  17) lfs-wrapper/0.0.1  20) rocm/6.0.0
  3) craype-network-ofi   6) cray-pmi/6.1.13                         9) cray-dsmml/0.2.2  12) PrgEnv-cray/8.5.0    15) darshan-runtime/3.4.0-mpi  18) DefApps

Cmake command:

$ cmake -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC ..

and then run `make.

mgates3 commented 1 month ago

An easy fix should be -DSCALAPACK_LIBRARIES=none. The testers won't be able to compare to a reference solution, but should otherwise work.

We don't actually need -lscalapack on Frontier, but I'm not sure the CMake syntax to skip it. Maybe -DSCALAPACK_LIBRARIES= (i.e., defining it as empty) would work.

devreal commented 1 month ago

Thanks @mgates3! I can confirm that -DSCALAPACK_LIBRARIES=none is a workaround. I can now build the tester on Frontier, woohoo!