search

ORNL / ReSolve

Library of GPU-resident linear solvers
Other
58 stars 2 forks source link

Building without `CMAKE_BUILD_TYPE` results in segaults #57

Closed cameronrutherford closed 8 months ago

cameronrutherford commented 1 year ago

This error was observed building and running on incline. Here is a sample error log, and CMakePresets.json used:

Build error log ``` [ruth521@deception resolve]$ export MY_CLUSTER=incline [ruth521@deception resolve]$ srun -A exasgd -p incline -t30 -N 1 --exclusive ./buildsystem/build.sh Paths: Source dir: /people/ruth521/projects/resolve Build dir: /people/ruth521/projects/resolve/build Install dir: /people/ruth521/projects/resolve/install Path to buildsystem script: /people/ruth521/projects/resolve/buildsystem/build.sh Currently Loaded Modulefiles: 1) gcc/8.4.0 2) rocm/5.3.0 3) curl/7.29.0-gcc-8.4.0-3emq5yx 4) gmake/4.4.1-gcc-8.4.0-l7nyr34 5) pkgconf/1.9.5-gcc-8.4.0-733ltud 6) ncurses/6.4-gcc-8.4.0-gwo76of 7) zlib-ng/2.1.4-gcc-8.4.0-feah6zt 8) cmake/3.27.7-gcc-8.4.0-rmou7zf 9) gmake/4.4.1-clang-16.0.0-rocm5.6.0-ku7zmdn 10) python/3.9.12-gcc-8.4.0-ob2n5zs 11) re2c/2.2-gcc-8.4.0-zmj4cst 12) ninja/1.11.1-gcc-8.4.0-ofxvwff 13) z3/4.11.2-gcc-8.4.0-363odap 14) llvm-amdgpu/5.6.1-gcc-8.4.0-vy3wrnq 15) rocm-core/5.6.1-gcc-8.4.0-llv2yv4 16) rocm-cmake/5.6.1-gcc-8.4.0-klwq5kk 17) comgr/5.6.1-gcc-8.4.0-yl7z2re 18) libiconv/1.17-gcc-8.4.0-o2hwfiz 19) diffutils/3.9-gcc-8.4.0-7ceszkk 20) bzip2/1.0.8-gcc-8.4.0-on73m5o 21) xz/5.4.1-gcc-8.4.0-v5kymdq 22) libxml2/2.10.3-gcc-8.4.0-6mgqxiy 23) pigz/2.7-gcc-8.4.0-btbzuey 24) zstd/1.5.5-gcc-8.4.0-3ets7dy 25) tar/1.34-gcc-8.4.0-atzwdgy 26) gettext/0.22.3-gcc-8.4.0-m33ujza 27) libsigsegv/2.14-gcc-8.4.0-gzna4n3 28) m4/1.4.19-gcc-8.4.0-bwzchwl 29) perl/5.26.0-gcc-8.4.0-6tdzqfd 30) bison/3.8.2-gcc-8.4.0-etuka2a 31) libmd/1.0.4-gcc-8.4.0-tbwjjiv 32) libbsd/0.11.7-gcc-8.4.0-tqu43be 33) expat/2.5.0-gcc-8.4.0-7rfhc6v 34) findutils/4.9.0-gcc-8.4.0-yjpuc44 35) flex/2.6.3-gcc-8.4.0-egfp6z5 36) util-macros/1.19.3-gcc-8.4.0-64inrmm 37) glproto/1.4.17-gcc-8.4.0-d2xwssq 38) libunwind/1.6.2-gcc-8.4.0-rgu2xlp 39) inputproto/2.3.2-gcc-8.4.0-cmbqp4h 40) kbproto/1.0.7-gcc-8.4.0-pq6oac7 41) libpthread-stubs/0.4-gcc-8.4.0-kcav646 42) xproto/7.0.31-gcc-8.4.0-2xcokuh 43) libxau/1.0.8-gcc-8.4.0-7r3byub 44) libxdmcp/1.1.4-gcc-8.4.0-ewgybb6 45) xcb-proto/1.15.2-gcc-8.4.0-kj2ch5l 46) libxcb/1.14-gcc-8.4.0-5c2iblj 47) xextproto/7.3.0-gcc-8.4.0-l75zwpb 48) xtrans/1.4.0-gcc-8.4.0-cbfrdws 49) libx11/1.8.4-gcc-8.4.0-n5i25lg 50) libxext/1.3.3-gcc-8.4.0-zr45mr7 51) libice/1.0.9-gcc-8.4.0-miq4xhk 52) util-linux-uuid/2.38.1-gcc-8.4.0-c2ehpex 53) libsm/1.2.3-gcc-8.4.0-4zhneq6 54) libxt/1.1.5-gcc-8.4.0-wxqofrn 55) py-pip/23.1.2-gcc-8.4.0-yajovh7 56) py-wheel/0.41.2-gcc-8.4.0-dkkw2va 57) py-setuptools/68.0.0-gcc-8.4.0-ihu4sfq 58) meson/1.2.2-gcc-8.4.0-vcdwjmb 59) py-markupsafe/2.1.3-gcc-8.4.0-ckndn5h 60) py-mako/1.2.4-gcc-8.4.0-djk4toh 61) renderproto/0.11.1-gcc-8.4.0-wf73fb4 62) libxrender/0.9.10-gcc-8.4.0-3rqjeli 63) randrproto/1.5.0-gcc-8.4.0-3ytqi3w 64) libxrandr/1.5.3-gcc-8.4.0-7ns64x7 65) xrandr/1.5.0-gcc-8.4.0-3whusyt 66) mesa/23.0.3-gcc-8.4.0-lbefc2e 67) glx/1.4-gcc-8.4.0-ly5ulwt 68) hipify-clang/5.6.1-gcc-8.4.0-e3jea5v 69) elfutils/0.189-gcc-8.4.0-23kjwto 70) libtool/2.4.7-gcc-8.4.0-2bmpsy4 71) libpciaccess/0.17-gcc-8.4.0-sh2c4la 72) libdrm/2.4.115-gcc-8.4.0-6h77lxh 73) autoconf/2.69-gcc-8.4.0-dcrbb7h 74) automake/1.16.5-gcc-8.4.0-tvi3cks 75) numactl/2.0.14-gcc-8.4.0-7mpcwqq 76) hsakmt-roct/5.6.1-gcc-8.4.0-4on3xib 77) hsa-rocr-dev/5.6.1-gcc-8.4.0-tdlpv7w 78) perl-file-which/1.27-gcc-8.4.0-nix64yx 79) perl-module-build/0.4232-gcc-8.4.0-ayed35p 80) perl-uri-encode/1.1.1-gcc-8.4.0-biqataj 81) py-ply/3.11-gcc-8.4.0-creftnl 82) py-cppheaderparser/2.7.4-gcc-8.4.0-nw7554i 83) rocminfo/5.6.1-gcc-8.4.0-5shaxxj 84) roctracer-dev-api/5.6.1-gcc-8.4.0-gbaoh25 85) hip/5.6.1-gcc-8.4.0-6agncv5 86) msgpack-c/3.1.1-gcc-8.4.0-buxbznu 87) procps/4.0.4-gcc-8.4.0-gyn6his 88) py-joblib/1.2.0-gcc-8.4.0-ukcd432 89) py-cython/0.29.36-gcc-8.4.0-5f4zyzb 90) py-msgpack/1.0.5-gcc-8.4.0-2xh5udm 91) libyaml/0.2.5-gcc-8.4.0-hidc7bw 92) py-pyyaml/6.0-gcc-8.4.0-4mdsdw2 93) py-distlib/0.3.7-gcc-8.4.0-f25ay4b 94) py-editables/0.3-gcc-8.4.0-hrmamrk 95) py-flit-core/3.9.0-gcc-8.4.0-q3yng6k 96) py-packaging/23.1-gcc-8.4.0-7krugqt 97) py-pathspec/0.11.1-gcc-8.4.0-vm5freh 98) git/2.42.0-gcc-8.4.0-k5crf2q 99) py-tomli/2.0.1-gcc-8.4.0-m4gh2nb 100) py-typing-extensions/4.8.0-gcc-8.4.0-ovqdpbs 101) py-setuptools-scm/7.1.0-gcc-8.4.0-hqzn5lb 102) py-pluggy/1.0.0-gcc-8.4.0-lqpf66l 103) py-calver/2022.6.26-gcc-8.4.0-pm6rj2c 104) py-trove-classifiers/2023.8.7-gcc-8.4.0-iy66qnh 105) py-hatchling/1.18.0-gcc-8.4.0-bjpjiiq 106) py-hatch-vcs/0.3.0-gcc-8.4.0-hc6rq3a 107) py-filelock/3.12.4-gcc-8.4.0-rzqmlrq 108) py-platformdirs/3.10.0-gcc-8.4.0-6hnyp7h 109) py-virtualenv/20.24.5-gcc-8.4.0-h4mzkzl 110) rocblas/5.6.1-gcc-8.4.0-6xcnn6z 111) rocprim/5.6.1-gcc-8.4.0-uypvsdi 112) rocsparse/5.6.1-gcc-8.4.0-irtacbe 113) gmp/6.2.1-gcc-8.4.0-jbyetgm 114) metis/5.1.0-gcc-8.4.0-arxpx4m 115) autoconf-archive/2023.02.20-gcc-8.4.0-peuq4u2 116) texinfo/7.0.3-gcc-8.4.0-eja2su5 117) mpfr/4.2.0-gcc-8.4.0-pf2mr55 118) openblas/0.3.24-gcc-8.4.0-lnpxyrn 119) suite-sparse/5.13.0-gcc-8.4.0-cy34eno ~/projects/resolve/build ~/projects/resolve Configuring Preset CMake variables: CMAKE_HIP_ARCHITECTURES="gfx908" CMAKE_INSTALL_PREFIX:PATH="/people/ruth521/projects/resolve/install" RESOLVE_USE_HIP="ON" -- The C compiler identification is Clang 16.0.0 -- The CXX compiler identification is Clang 16.0.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/llvm-amdgpu-5.6.1-vy3wrnqjafyyqecvn4azv2apududkipa/bin/clang - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/llvm-amdgpu-5.6.1-vy3wrnqjafyyqecvn4azv2apududkipa/bin/clang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found klu include: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/include -- Found klu library: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib/libklu.so -- Not using CUDA -- The HIP compiler identification is Clang 16.0.0 -- Detecting HIP compiler ABI info -- Detecting HIP compiler ABI info - done -- Check for working HIP compiler: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/llvm-amdgpu-5.6.1-vy3wrnqjafyyqecvn4azv2apududkipa/bin/clang++ - skipped -- Detecting HIP compile features -- Detecting HIP compile features - done CMake Deprecation Warning at /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake. Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. Call Stack (most recent call first): cmake/ReSolveFindHipLibraries.cmake:6 (find_package) CMakeLists.txt:87 (include) -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- hip::amdhip64 is SHARED_LIBRARY -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Success CMake Deprecation Warning at /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake. Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. Call Stack (most recent call first): /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/cmake-3.27.7-rmou7zfs7n3om4siyztdn3v4pze6aqqc/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package) /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib/cmake/rocblas/rocblas-config.cmake:90 (find_dependency) cmake/ReSolveFindHipLibraries.cmake:7 (find_package) CMakeLists.txt:87 (include) -- hip::amdhip64 is SHARED_LIBRARY -- HIP include directories: /qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/include -- Resolve libraries: -- Configuring done (11.2s) -- Generating done (1.7s) -- Build files have been written to: /people/ruth521/projects/resolve/build Building [ 2%] Building CXX object resolve/utilities/logger/CMakeFiles/resolve_logger.dir/Logger.cpp.o [ 4%] Linking CXX shared library libresolve_logger.so [ 4%] Built target resolve_logger [ 11%] Building HIP object resolve/hip/CMakeFiles/resolve_backend_hip.dir/MemoryUtils.hip.o [ 11%] Building HIP object resolve/hip/CMakeFiles/resolve_backend_hip.dir/hipKernels.hip.o [ 11%] Building HIP object resolve/hip/CMakeFiles/resolve_backend_hip.dir/hipVectorKernels.hip.o [ 13%] Linking HIP shared library libresolve_backend_hip.so [ 13%] Built target resolve_backend_hip [ 27%] Building CXX object resolve/workspace/CMakeFiles/resolve_workspace.dir/LinAlgWorkspaceHIP.cpp.o [ 27%] Building CXX object resolve/workspace/CMakeFiles/resolve_workspace.dir/LinAlgWorkspaceCpu.cpp.o [ 27%] Building CXX object resolve/vector/CMakeFiles/resolve_vector.dir/VectorHandlerCpu.cpp.o [ 27%] Building CXX object resolve/vector/CMakeFiles/resolve_vector.dir/Vector.cpp.o [ 27%] Building CXX object resolve/vector/CMakeFiles/resolve_vector.dir/VectorHandler.cpp.o [ 27%] Building CXX object resolve/vector/CMakeFiles/resolve_vector.dir/VectorHandlerHip.cpp.o [ 31%] Linking CXX shared library libresolve_vector.so [ 31%] Linking CXX shared library libresolve_workspace.so [ 31%] Built target resolve_workspace [ 31%] Built target resolve_vector [ 43%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/Coo.cpp.o [ 43%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/io.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/Sparse.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/MatrixHandlerHip.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/Csc.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/MatrixHandler.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/Csr.cpp.o [ 50%] Building CXX object resolve/matrix/CMakeFiles/resolve_matrix.dir/MatrixHandlerCpu.cpp.o [ 52%] Linking CXX shared library libresolve_matrix.so [ 52%] Built target resolve_matrix [ 56%] Building CXX object resolve/CMakeFiles/ReSolve.dir/LinSolver.cpp.o [ 56%] Building CXX object resolve/CMakeFiles/ReSolve.dir/LinSolverDirectKLU.cpp.o [ 59%] Linking CXX shared library libReSolve.so [ 59%] Built target ReSolve [ 68%] Building CXX object examples/CMakeFiles/klu_klu.exe.dir/r_KLU_KLU.cpp.o [ 77%] Building CXX object examples/CMakeFiles/klu_klu_standalone.exe.dir/r_KLU_KLU_standalone.cpp.o [ 79%] Building CXX object tests/functionality/CMakeFiles/klu_klu_test.exe.dir/testKLU.cpp.o [ 79%] Building CXX object tests/unit/matrix/CMakeFiles/runMatrixHandlerTests.exe.dir/runMatrixHandlerTests.cpp.o [ 79%] Building CXX object tests/unit/matrix/CMakeFiles/runMatrixIoTests.exe.dir/runMatrixIoTests.cpp.o [ 79%] Building CXX object tests/unit/vector/CMakeFiles/runGramSchmidtTests.exe.dir/runGramSchmidtTests.cpp.o [ 79%] Building CXX object tests/unit/vector/CMakeFiles/runVectorHandlerTests.exe.dir/runVectorHandlerTests.cpp.o [ 79%] Building CXX object tests/unit/memory/CMakeFiles/runMemoryUtilsTests.exe.dir/runMemoryUtilsTests.cpp.o [ 79%] Building CXX object tests/unit/utilities/logger/CMakeFiles/runLoggerTests.exe.dir/runLoggerTests.cpp.o [ 81%] Linking CXX executable runLoggerTests.exe [ 81%] Built target runLoggerTests.exe [ 84%] Linking CXX executable runMemoryUtilsTests.exe [ 86%] Linking CXX executable runMatrixIoTests.exe [ 86%] Built target runMemoryUtilsTests.exe [ 86%] Built target runMatrixIoTests.exe [ 88%] Linking CXX executable runMatrixHandlerTests.exe [ 90%] Linking CXX executable klu_klu_standalone.exe [ 95%] Linking CXX executable runGramSchmidtTests.exe [ 95%] Linking CXX executable klu_klu.exe [ 95%] Built target runMatrixHandlerTests.exe [100%] Linking CXX executable runVectorHandlerTests.exe [100%] Linking CXX executable klu_klu_test.exe [100%] Built target klu_klu_standalone.exe [100%] Built target runGramSchmidtTests.exe [100%] Built target klu_klu.exe [100%] Built target runVectorHandlerTests.exe [100%] Built target klu_klu_test.exe Installing [ 4%] Built target resolve_logger [ 13%] Built target resolve_backend_hip [ 20%] Built target resolve_workspace [ 31%] Built target resolve_vector [ 52%] Built target resolve_matrix [ 59%] Built target ReSolve [ 63%] Built target klu_klu.exe [ 68%] Built target klu_klu_standalone.exe [ 72%] Built target klu_klu_test.exe [ 77%] Built target runMatrixIoTests.exe [ 81%] Built target runMatrixHandlerTests.exe [ 86%] Built target runVectorHandlerTests.exe [ 90%] Built target runGramSchmidtTests.exe [ 95%] Built target runLoggerTests.exe [100%] Built target runMemoryUtilsTests.exe Install the project... -- Install configuration: "" -- Installing: /people/ruth521/projects/resolve/install/include/resolve/resolve_defs.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/utilities/logger/Logger.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/workspace/LinAlgWorkspace.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/workspace/LinAlgWorkspaceCpu.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/workspace/LinAlgWorkspaceCUDA.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/workspace/LinAlgWorkspaceHIP.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/vector/Vector.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/vector/VectorHandler.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/vector/VectorKernels.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/io.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/Sparse.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/Coo.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/Csr.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/Csc.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/matrix/MatrixHandler.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/hip/hipVectorKernels.h -- Installing: /people/ruth521/projects/resolve/install/include/resolve/hip/HipMemory.hpp -- Installing: /people/ruth521/projects/resolve/install/lib/libresolve_matrix.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libresolve_matrix.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/lib/libresolve_vector.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libresolve_vector.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/lib/libresolve_logger.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libresolve_logger.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib" -- Installing: /people/ruth521/projects/resolve/install/lib/libresolve_workspace.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libresolve_workspace.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/lib/libresolve_backend_hip.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libresolve_backend_hip.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/lib/libReSolve.so -- Set runtime path of "/people/ruth521/projects/resolve/install/lib/libReSolve.so" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/include/resolve/Common.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/cusolver_defs.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/LinSolver.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/LinSolverDirectCuSolverGLU.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/LinSolverDirectCuSolverRf.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/LinSolverDirectKLU.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/LinSolverIterativeFGMRES.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/RefactorizationSolver.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/SystemSolver.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/GramSchmidt.hpp -- Installing: /people/ruth521/projects/resolve/install/include/resolve/MemoryUtils.hpp -- Installing: /people/ruth521/projects/resolve/install/share/resolve/cmake/ReSolveTargets.cmake -- Installing: /people/ruth521/projects/resolve/install/share/resolve/cmake/ReSolveTargets-noconfig.cmake -- Installing: /people/ruth521/projects/resolve/install/share/resolve/cmake/ReSolveConfig.cmake -- Installing: /people/ruth521/projects/resolve/install/share/resolve/cmake/ReSolveConfigVersion.cmake -- Installing: /people/ruth521/projects/resolve/install/bin/klu_klu.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/klu_klu.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/klu_klu_standalone.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/klu_klu_standalone.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/share/examples/test.sh -- Installing: /people/ruth521/projects/resolve/install/share/examples/resolve_consumer -- Installing: /people/ruth521/projects/resolve/install/share/examples/resolve_consumer/CMakeLists.txt -- Installing: /people/ruth521/projects/resolve/install/share/examples/resolve_consumer/README.md -- Installing: /people/ruth521/projects/resolve/install/share/examples/resolve_consumer/consumer.cpp -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/klu_klu_test.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/klu_klu_test.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/matrix_ACTIVSg2000_AC_01.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg2000_AC_02.mtx.ones -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg200_AC_11.mtx.ones -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/matrix_ACTIVSg2000_AC_00.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg2000_AC_00.mtx.ones -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/matrix_ACTIVSg200_AC_10.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg2000_AC_00.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/matrix_ACTIVSg200_AC_11.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/read_mm.m -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg200_AC_10.mtx.ones -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg2000_AC_01.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg200_AC_11.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg200_AC_10.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/matrix_ACTIVSg2000_AC_02.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/functionality/data/rhs_ACTIVSg2000_AC_02.mtx -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMatrixIoTests.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMatrixIoTests.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMatrixHandlerTests.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMatrixHandlerTests.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runVectorHandlerTests.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runVectorHandlerTests.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runGramSchmidtTests.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runGramSchmidtTests.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" -- Installing: /people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMemoryUtilsTests.exe -- Set runtime path of "/people/ruth521/projects/resolve/install/bin/resolve/tests/unit/runMemoryUtilsTests.exe" to "/people/ruth521/projects/resolve/install/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/suite-sparse-5.13.0-cy34eno3dhhi3tavrf7o4qlk6mc6itpg/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocblas-5.6.1-6xcnn6zefeldwhba57vpy4jmvgu46sch/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/hip-5.6.1-6agncv5eyija6jdbr3kou6ojbeqxdqzf/lib:/qfs/projects/exasgd/resolve/spack-ci/install/linux-centos7-zen/gcc-8.4.0/rocsparse-5.6.1-irtacbe7nerhtmv3q2cojhhjxrruvmga/lib" ~/projects/resolve ~/projects/resolve/build ~/projects/resolve Testing Test project /people/ruth521/projects/resolve/build Start 1: klu_klu_test 1/7 Test #1: klu_klu_test ..................... Passed 0.45 sec Start 2: matrix_test 2/7 Test #2: matrix_test ...................... Passed 0.06 sec Start 3: matrix_handler_test 3/7 Test #3: matrix_handler_test ..............Subprocess aborted***Exception: 6.66 sec Memory access fault by GPU node-4 (Agent handle: 0x55935845d140) on address (nil). Reason: Page not present or supervisor privilege. Running tests on CPU: --- SKIP: Test matrixHandlerConstructor --- SKIP: Test matrixOneNorm --- PASS: Test matVec Running tests with HIP backend: --- SKIP: Test matrixHandlerConstructor --- SKIP: Test matrixOneNorm about to run mv Start 4: vector_handler_test ```
CMakePresets.json ``` { "version": 3, "cmakeMinimumRequired": { "major": 3, "minor": 22, "patch": 0 }, "configurePresets": [ { "name": "cuda", "displayName": "CUDA Build", "description": "Base config to build with CUDA", "binaryDir": "${sourceDir}/build", "installDir": "${sourceDir}/install", "generator": "Unix Makefiles", "cacheVariables": { "RESOLVE_USE_CUDA": "ON" } }, { "name": "rocm", "displayName": "ROCM build", "description": "Base config to build with ROCM", "binaryDir": "${sourceDir}/build", "installDir": "${sourceDir}/install", "generator": "Unix Makefiles", "cacheVariables": { "RESOLVE_USE_HIP": "ON" } }, { "name": "cpu", "displayName": "CPU only build", "description": "Base config to build without GPUs", "binaryDir": "${sourceDir}/build", "installDir": "${sourceDir}/install", "generator": "Unix Makefiles" }, { "name": "ascent", "inherits": "cuda", "displayName": "Ascent Build", "description": "Custom changes specific for Ascent", "cacheVariables": { "CMAKE_C_COMPILER": "$env{OLCF_GCC_ROOT}/bin/gcc", "CMAKE_CXX_COMPILER": "$env{OLCF_GCC_ROOT}/bin/g++" } }, { "name": "deception", "inherits": "cuda", "displayName": "Deception Build", "description": "Custom changes specific for Deception" }, { "name": "incline", "inherits": "rocm", "displayName": "Incline Build with rocm", "description": "Custom changes specific for Incline", "cacheVariables": { "CMAKE_HIP_ARCHITECTURES" : "gfx908" } } ] } ```

Not that simply using any other build type seems to fix the issue.

pelesh commented 1 year ago

@cameronrutherford, the incline build seems to work fine now. Has this issue been resolved?

cameronrutherford commented 1 year ago

@cameronrutherford, the incline build seems to work fine now. Has this issue been resolved?

This issue is still present. To clarify the issue description, I don't specify anything for CMAKE_BUILD_TYPE in the CMakePresets.json, and for some reason this results in a segfault at runtime. As soon as you try and build with a specific type, error goes away.

Two things that can help further debug this:

  1. Someone reproducing on a non-incline system
  2. Attempting to just pass CMAKE_CXX_FLAGS directly without a build type, and get debugging symbols into a failing build?

TBH this seems like a waste of time to debug, but I wanted to document it

pelesh commented 1 year ago

Removing "hackathon" milestone from this issue as it is marginal to the hackathon objectives.

pelesh commented 8 months ago

@cameronrutherford, so far I was not able to reproduce this issue on P9/V100, Milan/MI250X and Intel-based Macbook. I will close it as non-reproducible, but please feel free to reopen if the issue appears again or if you get more data.

cameronrutherford commented 8 months ago

Sounds good. I am hoping to never see this issue again ◡̈