search

spack / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
4.1k stars 2.19k forks source link

Installation issue: hipsolver #44805

Open Thomas-Ulrich opened 2 weeks ago

Thomas-Ulrich commented 2 weeks ago

Steps to reproduce the issue

ulrichth@uan02:~> spack spec -I  hipsolver@5.6.1%gcc@12.2.0~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
Input spec
--------------------------------
 -   hipsolver@5.6.1%gcc@12.2.0~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2

Concretized
--------------------------------
 -   hipsolver@5.6.1%gcc@12.2.0~cuda~ipo+rocm amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[^]      ^cmake@3.27.7%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen2
[^]          ^curl@8.4.0%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen2
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-sles15-zen2
[^]              ^openssl@3.1.3%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-sles15-zen2
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-sles15-zen2
[^]                  ^perl@5.38.0%gcc@12.2.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-sles15-zen2
[^]                      ^berkeley-db@18.1.40%gcc@12.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-zen2
[^]                      ^bzip2@1.0.8%gcc@12.2.0~debug~pic+shared build_system=generic arch=linux-sles15-zen2
[^]                          ^diffutils@3.9%gcc@12.2.0 build_system=autotools arch=linux-sles15-zen2
[^]                              ^libiconv@1.17%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-sles15-zen2
[^]                      ^gdbm@1.23%gcc@12.2.0 build_system=autotools arch=linux-sles15-zen2
[^]                          ^readline@8.2%gcc@12.2.0 build_system=autotools patches=bbf97f1 arch=linux-sles15-zen2
[^]              ^pkgconf@1.9.5%gcc@12.2.0 build_system=autotools arch=linux-sles15-zen2
[^]          ^ncurses@6.4%gcc@12.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-sles15-zen2
[^]          ^zlib-ng@2.1.4%gcc@12.2.0+compat+opt build_system=autotools arch=linux-sles15-zen2
[^]      ^gmake@4.4.1%gcc@12.2.0~guile build_system=generic arch=linux-sles15-zen2
[e]      ^hip@5.6.1%gcc@12.2.0~cuda+rocm build_system=cmake build_type=Release generator=make patches=aee7249,c2ee21c,e73e91b arch=linux-sles15-zen2
[e]      ^hsa-rocr-dev@5.6.1%gcc@12.2.0+image+shared build_system=cmake build_type=Release generator=make patches=9267179 arch=linux-sles15-zen2
[e]      ^llvm-amdgpu@5.6.1%gcc@12.2.0~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake build_type=Release generator=ninja patches=a08bbe1,b66529f,d35aec9 arch=linux-sles15-zen2
[e]      ^rocblas@5.6.1%gcc@12.2.0+tensile amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[e]      ^rocm-cmake@5.6.1%gcc@12.2.0 build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[e]      ^rocsolver@5.6.1%gcc@12.2.0+optimal amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2

Error message

Fatal Error: File '/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/llvm/include/iso_c_binding.mod' opened at (1) is not a GNU Fortran module file ==> Installing hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos [51/54] ==> No binary for hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos found: installing from source ==> Using cached archive: /project/project_465000831/spack_tandem/23.09/0.21.0/cache/_source-cache/archive/2e/2e546bc7771f7bf0aa7892b69cded725941573e8b70614759c3d03c21eb78dde.tar.gz ==> No patches needed for hipsolver ==> hipsolver: Executing phase: 'cmake' ==> hipsolver: Executing phase: 'build' ==> Error: ProcessError: Command exited with status 2: 'make' '-j20' 7 errors found in build log: 25 -- Looking for pthread_create in pthreads 26 -- Looking for pthread_create in pthreads - not found 27 -- Looking for pthread_create in pthread 28 -- Looking for pthread_create in pthread - found 29 -- Found Threads: TRUE 30 -- hip::amdhip64 is SHARED_LIBRARY >> 31 g++: error: unrecognized command-line option '--rtlib=compiler-rt' 32 CMake Deprecation Warning at /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib64/cmake/hip/hip-config.cmake:20 (cmake_minimum_required): 33 Compatibility with CMake < 3.5 will be removed from a future version of 34 CMake. 35 36 Update the VERSION argument value or use a ... suffix to tell 37 CMake that the project does not need compatibility with older versions. 38 Call Stack (most recent call first): 39 library/CMakeLists.txt:50 (find_package) 40 41 42 -- hip::amdhip64 is SHARED_LIBRARY >> 43 g++: error: unrecognized command-line option '--rtlib=compiler-rt' 44 CMake Deprecation Warning at /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib64/cmake/hip/hip-config.cmake:20 (cmake_minimum_required): 45 Compatibility with CMake < 3.5 will be removed from a future version of 46 CMake. 47 48 Update the VERSION argument value or use a ... suffix to tell 49 CMake that the project does not need compatibility with older versions. ... 51 /pfs/lustrep3/appl/lumi/spack/23.09/0.21.0/opt/spack/cmake-3.27.7-4amcecb/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package) 52 /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib64/cmake/rocblas/rocblas-config.cmake:90 (find_dependency) 53 library/src/CMakeLists.txt:104 (find_package) 54 55 56 -- hip::amdhip64 is SHARED_LIBRARY >> 57 g++: error: unrecognized command-line option '--rtlib=compiler-rt' 58 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY 59 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success 60 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY 61 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success 62 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR 63 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success ... 96 [ 75%] Building Fortran object library/src/CMakeFiles/hipsolver_fortran.dir/hipsolver_module.f90.o 97 cd /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g/library/src && /pfs/lustrep3/appl/lumi/spack/23.09/0.21.0-user/lib/spack/env/gcc/gfort ran -Dhipsolver_fortran_EXPORTS -O3 -DNDEBUG -O3 -J../../include/hipsolver/internal -fPIC -c /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-src/libra ry/src/hipsolver_module.f90 -o CMakeFiles/hipsolver_fortran.dir/hipsolver_module.f90.o 98 /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-src/library/src/hipsolver_module.f90:25:9: 99 100 25 | use iso_c_binding 101 | 1 >> 102 Fatal Error: File '/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/llvm/include/iso_c_binding.mod' opened at (1) is not a GNU Fortran module file 103 compilation terminated. >> 104 make[2]: *** [library/src/CMakeFiles/hipsolver_fortran.dir/build.make:78: library/src/CMakeFiles/hipsolver_fortran.dir/hipsolver_module.f90.o] Error 1 105 make[2]: Leaving directory '/tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g' >> 106 make[1]: *** [CMakeFiles/Makefile2:118: library/src/CMakeFiles/hipsolver_fortran.dir/all] Error 2 107 make[1]: *** Waiting for unfinished jobs.... 108 /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-src/library/src/amd_detail/hipsolver.cpp: In function 'rocblas_status hipsolverManageWorkspace(rocblas_ handle, size_t)': 109 /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-src/library/src/amd_detail/hipsolver.cpp:459:46: warning: 'bool rocblas_is_user_managing_device_memory( rocblas_handle)' is deprecated: "rocblas_is_user_managing_device_memory will be removed in a future release " "and supported modes will be 'rocblas_managed' & 'user_owned'" [-Wdeprecated-declarati ons] 110 459 | if(rocblas_is_user_managing_device_memory(handle)) 111 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~ 112 In file included from /pfs/lustrep3/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/include/rocblas/rocblas.h:33, ... 125 cd /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g/library/src && /pfs/lustrep3/appl/lumi/spack/23.09/0.21.0/opt/spack/cmake-3.27.7-4amce cb/bin/cmake -E cmake_link_script CMakeFiles/hipsolver.dir/link.txt --verbose=1 126 /pfs/lustrep3/appl/lumi/spack/23.09/0.21.0-user/lib/spack/env/gcc/g++ -fPIC -O3 -DNDEBUG -shared -Wl,-soname,libhipsolver.so.0 -o libhipsolver.so.0.1 CMakeFiles/hipsolver.dir/amd_detail/hipsolver_ conversions.cpp.o CMakeFiles/hipsolver.dir/amd_detail/hipsolver.cpp.o CMakeFiles/hipsolver.dir/amd_detail/hipsolver_compat.cpp.o CMakeFiles/hipsolver.dir/amd_detail/hipsolver_refactor.cpp.o CMakeF iles/hipsolver.dir/common/hipsolver_compat_common.cpp.o -Wl,-rpath,/pfs/lustrep3/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib:/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib:::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: /pfs/lustrep3/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib/librocsolver.so.0.1.50601 /pfs/lustrep3/appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib/ librocblas.so.3.0.50601 /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib/libamdhip64.so.5.6.50601 /opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0/libgcc.a 127 cd /tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g/library/src && /pfs/lustrep3/appl/lumi/spack/23.09/0.21.0/opt/spack/cmake-3.27.7-4amce cb/bin/cmake -E cmake_symlink_library libhipsolver.so.0.1 libhipsolver.so.0 libhipsolver.so 128 make[2]: Leaving directory '/tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g' 129 [ 87%] Built target hipsolver 130 make[1]: Leaving directory '/tmp/ulrichth/spack-stage/spack-stage-hipsolver-5.6.1-6ilkv4gahpre5can6h7pth4aqpyhzaos/spack-build-6ilkv4g' >> 131 make: *** [Makefile:159: all] Error 2

Information on your system

It is LUMI (euroHPC)

Additional information

@cgmb @renjithravindrankannath @srekolam spack-build-env.txt spack-build-out.txt

General information

cgmb commented 1 week ago

It looks like you might be getting a mix of the flang toolchain (from /opt/rocm) and the gfortran toolchain (from /usr). If that is what is happening, then as a workaround, you could potentially try setting the /opt/rocm flang as your Fortran compiler when building hipsolver.

Thomas-Ulrich commented 1 week ago

Thank you! It seems the compilers are correctly set:

ulrichth@uan02:/project/project_465000831/spack_tandem> spack compiler info rocmcc
rocmcc@5.6.1:
        paths:
                cc = /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/bin/amdclang
                cxx = /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/bin/amdclang++
                f77 = /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/bin/amdflang
                fc = /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/bin/amdflang
        environment:
            set:
                RFE_811452_DISABLE = 1
        Extra rpaths:
                /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib
                /appl/lumi/SW/CrayEnv/EB/rocm/5.6.1/lib64
                /opt/cray/pe/gcc-libs
        modules  = []
        operating system  = sles15
rocmcc@5.2.3:
        paths:
                cc = /opt/rocm-5.2.3/bin/amdclang
                cxx = /opt/rocm-5.2.3/bin/amdclang++
                f77 = /opt/rocm-5.2.3/bin/amdflang
                fc = /opt/rocm-5.2.3/bin/amdflang
        modules  = []
        operating system  = sles15
ulrichth@uan02:/project/project_465000831/spack_tandem> spack compiler info gcc
gcc@11.2.0:
        paths:
                cc = /opt/cray/pe/gcc/11.2.0/bin/gcc
                cxx = /opt/cray/pe/gcc/11.2.0/bin/g++
                f77 = /opt/cray/pe/gcc/11.2.0/bin/gfortran
                fc = /opt/cray/pe/gcc/11.2.0/bin/gfortran
        modules  = []
        operating system  = sles15
gcc@12.2.0:
        paths:
                cc = /opt/cray/pe/gcc/12.2.0/bin/gcc
                cxx = /opt/cray/pe/gcc/12.2.0/bin/g++
                f77 = /opt/cray/pe/gcc/12.2.0/bin/gfortran
                fc = /opt/cray/pe/gcc/12.2.0/bin/gfortran
        modules  = []
        operating system  = sles15
gcc@7.5.0:
        paths:
                cc = /usr/bin/gcc
                cxx = /usr/bin/g++
                f77 = /usr/bin/gfortran
                fc = /usr/bin/gfortran
        modules  = []
        operating system  = sles15

So you would suggest trying to change fc in compilers.yaml for gcc@12.2.0, just for this install?

cgmb commented 1 week ago

Yes. If that works, it's a good indicator my analysis is correct, though I would only suggest it as a temporary workaround until the root cause can be fixed. I expect that a better solution could be found, but I am travelling and not able to investigate further.

When I return in July, I will introduce a build option into hipsolver for disabling the Fortran component. That is not a real fix either, but it would have been another useful workaround. It's already disabled for Windows builds, so it would be trivial to make optional for Linux.