search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.19k stars 559 forks source link

KokkosKernels: Enabling TPL ROCSPARSE causes a compiler error #12296

Closed jjellio closed 8 months ago

jjellio commented 10 months ago

Bug Report

I've tried setting TPL_ENABLE_ROCSPARSE=ON with Trilinos/develop, and this is causing a compiler error from KK. This is an MI250x build (HIP execution space, etc..) This is also a COMPLEX_DOUBLE enabled build

[338/3336] Building CXX object packages/kokkos-kernels/CMakeFiles/kokkoskern...DINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o
FAILED: packages/kokkos-kernels/CMakeFiles/kokkoskernels.dir/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o 
/opt/cray/pe/craype/2.7.23/bin/CC -DKOKKOS_DEPENDENCE -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -Dkokkoskernels_EXPORTS -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/blas -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/blas -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/graph -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/graph -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/ode -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/common/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/common/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/common/unit_test -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/batched/eti -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/dense/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/dense/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/dense/unit_test -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/sparse/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/sparse/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/batched/sparse/unit_test -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/blas/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/blas/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/blas/eti -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/blas/eti -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/blas/tpls -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/graph/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/graph/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/graph/eti -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/graph/eti -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/eti -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/sparse/eti -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/ode/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/ode/impl -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/ode/unit_test -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos/core/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos/core/src -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos/core/src/../../tpls/desul/include -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos/containers/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos/containers/src -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos/algorithms/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos/algorithms/src -I/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos/simd/src -I/g/g20/jjellio/src/github/Trilinos/packages/kokkos/simd/src -isystem /p/lustre1/jjellio/spack/install/linux-rhel8-zen3/rocmcc-5.6.1/openblas-0.3.23-oujz2g3rban7idqhsihbkzo7czlbujjw/include -isystem /p/lustre1/jjellio/spack/install/linux-rhel8-zen3/rocmcc-5.6.1/metis-5.1.0-vkguyliuru34d2okfonvguxbang3ezvv/include -x hip -mllvm -amdgpu-early-inline-all=false -mllvm -amdgpu-function-calls=false -g -march=znver3 -mtune=znver3 -fno-gpu-rdc -xhip --rocm-path=/opt/rocm-5.6.1 --offload-arch=gfx90a  -O3 -DNDEBUG -std=c++17 -fPIC -march=znver3 -mtune=znver3 -fno-gpu-rdc -xhip --rocm-path=/opt/rocm-5.6.1 --offload-arch=gfx90a -MD -MT packages/kokkos-kernels/CMakeFiles/kokkoskernels.dir/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o -MF packages/kokkos-kernels/CMakeFiles/kokkoskernels.dir/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o.d -o packages/kokkos-kernels/CMakeFiles/kokkoskernels.dir/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o -c /tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp
In file included from /tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos/packages/kokkos-kernels/sparse/eti/generated_specializations_cpp/gmres/Sparse_gmres_eti_DOUBLE_ORDINAL_INT_OFFSET_SIZE_T_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp:23:
In file included from /g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/impl/KokkosSparse_gmres_spec.hpp:30:
In file included from /g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/impl/KokkosSparse_gmres_impl.hpp:30:
In file included from /g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/src/KokkosSparse_spmv.hpp:28:
In file included from /g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/impl/KokkosSparse_spmv_bsrmatrix_spec.hpp:474:
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const float, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1045:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const float, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const float *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<float *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(float, rocsparse_int, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const float, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1048:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const float, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const float *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<float *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(float, rocsparse_int, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const double, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1051:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const double, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const double *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<double *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(double, rocsparse_int, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const double, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1054:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const double, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const double *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<double *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(double, rocsparse_int, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<float>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1057:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<float>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const Kokkos::complex<float> *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<Kokkos::complex<float> *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(Kokkos::complex<float>, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<float>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1060:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<float>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const Kokkos::complex<float> *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<Kokkos::complex<float> *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(Kokkos::complex<float>, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<double>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1064:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<double>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const Kokkos::complex<double> *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<Kokkos::complex<double> *, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(Kokkos::complex<double>, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917:40: error: no type named 'block_layout' in 'KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<double>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>'
      std::is_same_v<typename AMatrix::block_layout, Kokkos::LayoutRight>,
                     ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1067:1: note: in instantiation of function template specialization 'KokkosSparse::Experimental::Impl::spmv_block_impl_rocsparse<KokkosSparse::Experimental::BsrMatrix<const Kokkos::complex<double>, const int, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>, const int>, Kokkos::View<const Kokkos::complex<double> *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<3>>, Kokkos::View<Kokkos::complex<double> *, Kokkos::LayoutRight, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, Kokkos::MemoryTraits<1>>>' requested here
KOKKOSSPARSE_SPMV_ROCSPARSE(Kokkos::complex<double>, rocsparse_int,
^
/g/g20/jjellio/src/github/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:1040:7: note: expanded from macro 'KOKKOSSPARSE_SPMV_ROCSPARSE'
      spmv_block_impl_rocsparse(controls, mode, alpha, A, x, beta, y);         \
      ^
8 errors generated when compiling for gfx90a.

My configure script isn't going to be too helpful, unless you can acces the system I am on. This should be a matter of just trying to configure Trilinos on an AMD system with -DTPL_ENABLE_ROCSPARSE=ON

jhux2 commented 10 months ago

@trilinos/kokkos-kernels

lucbv commented 10 months ago

@jjellio could you still give us the part of your configuration associated with Kokkos and Kokkos Kernels? That would make it easier for us to reproduce this.

jjellio commented 10 months ago

Seems like you just need to enable -DTPL_ENABLE_ROCSPARSE=ON ...

But I've cut down the configure to effectively be just that:

Cmake ``` cmake \ "-GNinja" \ "-DCMAKE_BUILD_TYPE:STRING=Release" \ "-DBUILD_SHARED_LIBS:BOOL=ON" \ "-DTrilinos_ENABLE_KokkosKernels=ON" \ "-DTPL_ENABLE_ROCSPARSE=ON" \ "-DTPL_ENABLE_ROCBLAS=ON" \ "-DTrilinos_ENABLE_Kokkos=ON" \ "-DKokkos_ENABLE_HIP:BOOL=ON" \ "-DKokkos_ARCH_VEGA90A:BOOL=ON" \ "-DCMAKE_EXE_LINKER_FLAGS=-x none --hip-link " \ "-DCMAKE_SHARED_LINKER_FLAGS=-x none --hip-link " \ "-DCMAKE_CXX_COMPILER=amdclang++" \ "-DCMAKE_C_COMPILER=amdclang" \ "-DCMAKE_CXX_FLAGS=-x hip -mllvm -amdgpu-early-inline-all=false -mllvm -amdgpu-function-calls=false -g " \ "-DCMAKE_CXX_STANDARD=17" \ "-DTrilinos_ENABLE_Fortran:BOOL=OFF" \ "-DTrilinos_ENABLE_COMPLEX_DOUBLE:BOOL=ON" \ "-DTPL_ENABLE_BinUtils:BOOL=OFF" \ "-DBinUtils_INCLUDE_DIRS=/include" \ "-DBinUtils_LIBRARY_DIRS=/lib" \ "-DTPL_ENABLE_BLAS:BOOL=ON" \ "-DTPL_ENABLE_LAPACK:BOOL=ON" \ "-DBLAS_LIBRARY_NAMES=sci_amd" \ "-DBLAS_INCLUDE_DIRS=/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/include" \ "-DBLAS_LIBRARY_DIRS=/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib" \ "-DLAPACK_LIBRARY_NAMES=sci_amd" \ "-DLAPACK_INCLUDE_DIRS=/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/include" \ "-DLAPACK_LIBRARY_DIRS=/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib" \ "-DTPL_ENABLE_Boost:BOOL=OFF" \ "-DTPL_ENABLE_BoostLib:BOOL=OFF" \ "-DTPL_ENABLE_METIS:BOOL=OFF" \ "-DTPL_ENABLE_ParMETIS:BOOL=OFF" \ "-DTPL_ENABLE_CGNS:BOOL=OFF" \ "-DTPL_ENABLE_HDF5:BOOL=OFF" \ "-DTPL_ENABLE_Netcdf:BOOL=OFF" \ "-DTPL_ENABLE_SuperLUDist:BOOL=OFF" \ "-DTPL_ENABLE_Matio=OFF" \ "-DTPL_ENABLE_X11=OFF" \ "-DTPL_ENABLE_MPI:BOOL=ON" \ "-DMPI_USE_COMPILER_WRAPPERS=OFF" \ "-DTPL_ENABLE_DLlib:BOOL=ON" \ "-DDLlib_INCLUDE_DIRS=/opt/rocm-5.6.1/include" \ "-DDLlib_LIBRARY_DIRS=/opt/rocm-5.6.1/lib" \ "-DDLlib_LIBRARY_NAMES=dl;hipsolver;rocsolver;hipblas;rocblas;hipsparse;rocsparse;amd_comgr;hsa-runtime64;amdhip64" \ "-DCMAKE_INSTALL_PREFIX=/p/lustre1/jjellio/installs/rzvernal/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly" \ /tmp/jjellio/Trilinos ```
Configure Output ``` [jjellio@rzvernal25 15:21 trilinos]$ ../my_configure.sh -- The Current CXX Standard is : 17 Configuring Trilinos build directory -- PROJECT_SOURCE_DIR='/tmp/jjellio/Trilinos' -- PROJECT_BINARY_DIR='/tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos' -- Trilinos_TRIBITS_DIR='/tmp/jjellio/Trilinos/cmake/tribits' -- TriBITS_VERSION_STRING='0.9 (Dev)' -- CMAKE_VERSION='3.27.4' -- CMAKE_GENERATOR='Ninja' -- CMAKE_HOST_SYSTEM_NAME='Linux' -- Trilinos_HOSTNAME='rzvernal25' CMake Warning (dev) at cmake/tribits/core/package_arch/TribitsFindPythonInterp.cmake:64 (find_package): Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules are removed. Run "cmake --help-policy CMP0148" for policy details. Use the cmake_policy command to set the policy and suppress this warning. Call Stack (most recent call first): cmake/tribits/core/package_arch/TribitsFindPythonInterp.cmake:80 (tribits_find_python) cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:107 (tribits_find_python_interp) cmake/tribits/core/package_arch/TribitsProject.cmake:92 (tribits_project_impl) CMakeLists.txt:103 (TRIBITS_PROJECT) This warning is for project developers. Use -Wno-dev to suppress it. -- Found PythonInterp: /opt/cray/pe/python/3.9.13.1/bin/python (found suitable version "3.9.13", minimum required is "2.6") -- PYTHON_EXECUTABLE='/opt/cray/pe/python/3.9.13.1/bin/python' -- Found Git: /usr/tce/bin/git (found version "2.29.1") Setting up major user options ... -- Setting Trilinos_ENABLE_TriKota=OFF because '/tmp/jjellio/Trilinos/packages/TriKota/Dakota' does not exist! -- Trilinos_GENERATE_REPO_VERSION_FILE='ON' Trilinos repos versions: -------------------------------------------------------------------------------- *** Base Git Repo: Trilinos 87d0c5c27f7 [Fri Sep 22 11:17:04 2023 -0600] Intrepid2: add hierarchical bases for H(div) pyramids (#12286) -------------------------------------------------------------------------------- Reading list of native TPLs from /tmp/jjellio/Trilinos/TPLsList.cmake -- After reading above TPLsList.cmake file: Trilinos_NUM_DEFINED_TPLS='122' Reading list of native packages from /tmp/jjellio/Trilinos/PackagesList.cmake -- After reading above PackagesList.cmake file: Trilinos_NUM_DEFINED_INTERNAL_TOPLEVEL_PACKAGES='59' Processing Project, Repository, and Package dependency files and building internal dependencies graph ... -- NOTE: Setting Trilinos_ENABLE_SEACASMapvarlib=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASExodus_for=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASExoIIv2for32=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASSupes=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASSuplib=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASPLT=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASBlot=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASFastq=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASSVDI=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASAlgebra=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASExotxt=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASGjoin=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASGen3D=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASGenshell=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASGrepos=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASExplore=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASMapvar=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASMapvar-kd=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASNumbers=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASTxtexo=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASEx2ex1v2=OFF because Trilinos_ENABLE_Fortran=OFF! -- NOTE: Setting Trilinos_ENABLE_SEACASEx1ex2v2=OFF because Trilinos_ENABLE_Fortran=OFF! -- Trilinos_NUM_DEFINED_INTERNAL_PACKAGES='151' Package dependencies information: -- Trilinos_NUM_DEFINED_TPLS='122' -- Trilinos_NUM_DEFINED_INTERNAL_TOPLEVEL_PACKAGES='59' -- Trilinos_NUM_DEFINED_TOPLEVEL_PACKAGES='181' -- Trilinos_NUM_DEFINED_INTERNAL_PACKAGES='151' -- Trilinos_NUM_DEFINED_PACKAGES='273' Explicitly enabled top-level packages on input (by user): Kokkos KokkosKernels 2 Explicitly enabled packages on input (by user): Kokkos KokkosKernels 2 Explicitly disabled top-level packages on input (by user or by default): NewPackage 1 Explicitly disabled packages on input (by user or by default): ShyLU_NodeBasker ShyLU_NodeFastILU SEACASExodus_for SEACASExoIIv2for32 SEACASSupes SEACASSuplib SEACASSVDI SEACASPLT SEACASAlgebra SEACASBlot SEACASExotxt SEACASEx1ex2v2 SEACASExotec2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASExplore SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASNumbers SEACASTxtexo SEACASEx2ex1v2 STKMiddle_mesh STKSearchUtil STKTransferUtil PanzerExprEval NewPackage 30 Explicitly enabled external packages/TPLs on input (by user): ROCBLAS ROCSPARSE MPI BLAS LAPACK DLlib 6 Explicitly disabled external packages/TPLs on input (by user or by default): BinUtils Boost METIS ParMETIS HDF5 CGNS Netcdf SuperLUDist BoostLib Matio X11 11 Disabling all packages that have a required dependency on disabled TPLs and optional package TPL support based on TPL_ENABLE_=OFF ... Disabling subpackages for hard disables of parent packages due to Trilinos_ENABLE_=OFF ... Disabling forward required packages and optional intra-package support that have a dependency on disabled packages Trilinos_ENABLE_=OFF (or TPL_ENABLE_=OFF) ... -- Setting Trilinos_ENABLE_MiniTensor=OFF because MiniTensor has a required library dependence on disabled package Boost -- Setting Trilinos_ENABLE_Krino=OFF because Krino has a required library dependence on disabled package Boost -- Setting KokkosKernels_ENABLE_METIS=OFF because KokkosKernels has an optional library dependence on disabled package METIS -- Setting Trilinos_ENABLE_SEACASExodus=OFF because SEACASExodus has a required library dependence on disabled package Netcdf -- Setting Trilinos_ENABLE_SEACASNemesis=OFF because SEACASNemesis has a required library dependence on disabled package Netcdf -- Setting Trilinos_ENABLE_SEACASExo2mat=OFF because SEACASExo2mat has a required library dependence on disabled package Matio -- Setting Trilinos_ENABLE_SEACASMat2exo=OFF because SEACASMat2exo has a required library dependence on disabled package Matio -- Setting Trilinos_ENABLE_SEACASConjoin=OFF because SEACASConjoin has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASEjoin=OFF because SEACASEjoin has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASEpu=OFF because SEACASEpu has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASCpup=OFF because SEACASCpup has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASExodiff=OFF because SEACASExodiff has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASExo_format=OFF because SEACASExo_format has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASNas2exo=OFF because SEACASNas2exo has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASZellij=OFF because SEACASZellij has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASNemslice=OFF because SEACASNemslice has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASNemspread=OFF because SEACASNemspread has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACASSlice=OFF because SEACASSlice has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_SEACAS=OFF because SEACAS has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_STKIO=OFF because STKIO has a required library dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_STKBalance=OFF because STKBalance has a required library dependence on disabled package SEACASExodus -- Setting PanzerAdaptersSTK_ENABLE_TESTS=OFF because PanzerAdaptersSTK has a required test/example dependence on disabled package SEACASExodus -- Setting PanzerAdaptersSTK_ENABLE_EXAMPLES=OFF because PanzerAdaptersSTK has a required test/example dependence on disabled package SEACASExodus -- Setting Trilinos_ENABLE_STKTools=OFF because STKTools has a required library dependence on disabled package STKIO -- Setting Trilinos_ENABLE_Percept=OFF because Percept has a required library dependence on disabled package STKIO -- Setting Trilinos_ENABLE_PanzerAdaptersSTK=OFF because PanzerAdaptersSTK has a required library dependence on disabled package STKIO -- Setting Trilinos_ENABLE_PanzerMiniEM=OFF because PanzerMiniEM has a required library dependence on disabled package PanzerAdaptersSTK Enabling subpackages for hard enables of parent packages due to Trilinos_ENABLE_=ON ... Disabling subpackage tests/examples based on parent package tests/examples disables ... Enabling subpackage tests/examples based on parent package tests/examples enables ... Enabling all required (and optional since Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=ON) upstream packages for current set of enabled packages (Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON) ... Enabling all optional intra-package enables _ENABLE_ that are not currently disabled if both sets of packages are enabled ... -- NOT setting Kokkos_ENABLE_Pthread=ON since Pthread is NOT enabled at this point! -- NOT setting Kokkos_ENABLE_CUDA=ON since CUDA is NOT enabled at this point! -- NOT setting Kokkos_ENABLE_HWLOC=ON since HWLOC is NOT enabled at this point! -- Setting Kokkos_ENABLE_DLlib=ON since Trilinos_ENABLE_Kokkos=ON AND TPL_ENABLE_DLlib=ON -- NOT setting Kokkos_ENABLE_CUSPARSE=ON since CUSPARSE is NOT enabled at this point! -- Setting KokkosKernels_ENABLE_Kokkos=ON since Trilinos_ENABLE_KokkosKernels=ON AND Trilinos_ENABLE_Kokkos=ON -- NOT setting KokkosKernels_ENABLE_quadmath=ON since quadmath is NOT enabled at this point! -- NOT setting KokkosKernels_ENABLE_MKL=ON since MKL is NOT enabled at this point! -- Setting KokkosKernels_ENABLE_BLAS=ON since Trilinos_ENABLE_KokkosKernels=ON AND TPL_ENABLE_BLAS=ON -- Setting KokkosKernels_ENABLE_LAPACK=ON since Trilinos_ENABLE_KokkosKernels=ON AND TPL_ENABLE_LAPACK=ON -- NOT setting KokkosKernels_ENABLE_CUSPARSE=ON since CUSPARSE is NOT enabled at this point! -- NOT setting KokkosKernels_ENABLE_SuperLU=ON since SuperLU is NOT enabled at this point! -- NOT setting KokkosKernels_ENABLE_Cholmod=ON since Cholmod is NOT enabled at this point! -- NOT setting KokkosKernels_ENABLE_CUBLAS=ON since CUBLAS is NOT enabled at this point! -- Setting KokkosKernels_ENABLE_ROCBLAS=ON since Trilinos_ENABLE_KokkosKernels=ON AND TPL_ENABLE_ROCBLAS=ON -- Setting KokkosKernels_ENABLE_ROCSPARSE=ON since Trilinos_ENABLE_KokkosKernels=ON AND TPL_ENABLE_ROCSPARSE=ON -- NOT setting KokkosKernels_ENABLE_yaml-cpp=ON since yaml-cpp is NOT enabled at this point! Set cache entries for optional packages/TPLs and tests/examples for packages actually enabled ... Enabling the shell of non-enabled parent packages (mostly for show) that have at least one subpackage enabled ... Adjust the set of internal and external packages: Final set of enabled top-level packages: Kokkos KokkosKernels 2 Final set of enabled packages: Kokkos KokkosKernels 2 Final set of non-enabled top-level packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest Teuchos RTOp Sacado MiniTensor Epetra Zoltan Shards Triutils EpetraExt Tpetra TrilinosSS Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Anasazi Ifpack2 Stratimikos Teko Intrepid Intrepid2 Compadre STK Percept Krino Phalanx NOX MueLu Zoltan2Sphynx Zoltan2 ShyLU_DD ShyLU Tempus Stokhos ROL Piro Panzer PyTrilinos NewPackage Adelus TrilinosCouplings TrilinosBuildStats TrilinosInstallTests 57 Final set of non-enabled packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest TeuchosCore TeuchosParser TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos RTOp Sacado MiniTensor Epetra Zoltan Shards Triutils EpetraExt TpetraTSQR TpetraCore Tpetra TrilinosSS ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU ShyLU_Node Amesos2 SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASNemesis SEACASIoss SEACASChaco SEACASAprepro_lib SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASPLT SEACASAlgebra SEACASAprepro SEACASBlot SEACASConjoin SEACASEjoin SEACASEpu SEACASCpup SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExotxt SEACASExo_format SEACASEx1ex2v2 SEACASExotec2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASExplore SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASMat2exo SEACASNas2exo SEACASZellij SEACASNemslice SEACASNemspread SEACASNumbers SEACASSlice SEACASTxtexo SEACASEx2ex1v2 SEACAS Anasazi Ifpack2 Stratimikos Teko Intrepid Intrepid2 Compadre STKUtil STKCoupling STKMath STKSimd STKNGP_TEST STKExprEval STKTopology STKSearch STKMiddle_mesh STKTransfer STKMesh STKIO STKTools STKBalance STKUnit_test_utils STKSearchUtil STKTransferUtil STKUnit_tests STKDoc_tests STKEmend STK Percept Krino Phalanx NOX MueLu Zoltan2Sphynx Zoltan2 ShyLU_DDFROSch ShyLU_DDCore ShyLU_DDCommon ShyLU_DD ShyLU Tempus Stokhos ROL Piro PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK PanzerMiniEM PanzerExprEval Panzer PyTrilinos NewPackage Adelus TrilinosCouplings TrilinosBuildStats TrilinosInstallTests 149 Final set of enabled top-level external packages/TPLs: ROCBLAS ROCSPARSE MPI BLAS LAPACK DLlib 6 Final set of enabled external packages/TPLs: ROCBLAS ROCSPARSE MPI BLAS LAPACK DLlib 6 Final set of non-enabled top-level external packages/TPLs: MKL yaml-cpp Peano CUDA CUBLAS CUSOLVER CUSPARSE Thrust Cusp TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Boost Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf Netcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 116 Final set of non-enabled external packages/TPLs: MKL yaml-cpp Peano CUDA CUBLAS CUSOLVER CUSPARSE Thrust Cusp TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Boost Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf Netcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 116 Setting up export dependencies for all enabled packages ... Probing the environment ... -- USE_XSDK_DEFAULTS='FALSE' -- BUILD_SHARED_LIBS='ON' -- CMAKE_BUILD_TYPE='Release' -- MPI_USE_COMPILER_WRAPPERS='OFF' -- MPI_EXEC='MPI_EXEC-NOTFOUND' -- The C compiler identification is Clang 16.0.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /opt/rocm-5.6.1/bin/amdclang - skipped -- Detecting C compile features -- Detecting C compile features - done -- CMAKE_C_COMPILER_ID='Clang' -- CMAKE_C_COMPILER_VERSION='16.0.0' -- The CXX compiler identification is Clang 16.0.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /opt/rocm-5.6.1/bin/amdclang++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- CMAKE_CXX_COMPILER_ID='Clang' -- CMAKE_CXX_COMPILER_VERSION='16.0.0' -- Skip adding flags for OpenMP because Kokkos flags does that ... -- Trilinos_SET_INSTALL_RPATH='TRUE' -- CMAKE_INSTALL_RPATH_USE_LINK_PATH='TRUE' -- Setting default for CMAKE_INSTALL_RPATH pointing to Trilinos_INSTALL_LIB_DIR -- CMAKE_INSTALL_RPATH='/p/lustre1/jjellio/installs/rzvernal/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/lib64' -- Looking for C++ include sys/time.h -- Looking for C++ include sys/time.h - found -- Looking for C++ include time.h -- Looking for C++ include time.h - found -- Looking for C++ include stdint.h -- Looking for C++ include stdint.h - found -- Looking for C++ include inttypes.h -- Looking for C++ include inttypes.h - found -- Found Perl: /usr/bin/perl (found version "5.26.3") -- Performing Test MATH_LIBRARY_IS_SUPPLIED -- Performing Test MATH_LIBRARY_IS_SUPPLIED - Failed -- Appending math library /usr/lib64/libm.so to link line ... -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN - Failed -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN - Success -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF - Failed -- Performing Test FINITE_VALUE_HAVE_STD_ISINF -- Performing Test FINITE_VALUE_HAVE_STD_ISINF - Success -- Found Doxygen: /usr/bin/doxygen (found version "1.8.14") found components: doxygen dot Getting information for all enabled external packages/TPLs ... Processing enabled external package/TPL: ROCBLAS (enabled explicitly, disable with -DTPL_ENABLE_ROCBLAS=OFF) CMake Deprecation Warning at /opt/rocm-5.6.1/hip/lib/cmake/hip/hip-config.cmake:20 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake. Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. Call Stack (most recent call first): /p/lustre1/jjellio/spack/install/linux-rhel8-zen3/rocmcc-5.6.1/cmake-3.27.4-h3y7pd5bqvtstmoqb6na5usxgwu2fz2e/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package) /opt/rocm-5.6.1/lib/cmake/rocblas/rocblas-config.cmake:90 (find_dependency) cmake/TPLs/FindTPLROCBLAS.cmake:56 (find_package) cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:226 (include) cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:170 (tribits_process_enabled_tribits_find_tpl_mod_file) cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:105 (tribits_process_enabled_standard_tpl) cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:197 (tribits_process_enabled_tpls) cmake/tribits/core/package_arch/TribitsProject.cmake:92 (tribits_project_impl) CMakeLists.txt:103 (TRIBITS_PROJECT) -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- hip::amdhip64 is SHARED_LIBRARY -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS -- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Success Processing enabled external package/TPL: ROCSPARSE (enabled explicitly, disable with -DTPL_ENABLE_ROCSPARSE=OFF) Processing enabled external package/TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF) -- MPI_LIBRARY_NAMES='' Processing enabled external package/TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF) -- BLAS_LIBRARY_NAMES='sci_amd' -- Searching for libs in BLAS_LIBRARY_DIRS='/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib' -- Searching for a lib in the set "sci_amd": -- Searching for lib 'sci_amd' ... -- Found lib '/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib/libsci_amd.so' -- TPL_BLAS_LIBRARIES='/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib/libsci_amd.so' Processing enabled external package/TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF) -- LAPACK_LIBRARY_NAMES='sci_amd' -- Searching for libs in LAPACK_LIBRARY_DIRS='/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib' -- Searching for a lib in the set "sci_amd": -- Searching for lib 'sci_amd' ... -- Found lib '/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib/libsci_amd.so' -- TPL_LAPACK_LIBRARIES='/opt/cray/pe/libsci/23.09.1.1/AMD/5.0/x86_64/lib/libsci_amd.so' Processing enabled external package/TPL: DLlib (enabled explicitly, disable with -DTPL_ENABLE_DLlib=OFF) -- DLlib_LIBRARY_NAMES='dl;hipsolver;rocsolver;hipblas;rocblas;hipsparse;rocsparse;amd_comgr;hsa-runtime64;amdhip64' -- Searching for libs in DLlib_LIBRARY_DIRS='/opt/rocm-5.6.1/lib' -- Searching for a lib in the set "dl": -- Searching for lib 'dl' ... -- Found lib '/usr/lib64/libdl.so' -- Searching for a lib in the set "hipsolver": -- Searching for lib 'hipsolver' ... -- Found lib '/opt/rocm-5.6.1/lib/libhipsolver.so' -- Searching for a lib in the set "rocsolver": -- Searching for lib 'rocsolver' ... -- Found lib '/opt/rocm-5.6.1/lib/librocsolver.so' -- Searching for a lib in the set "hipblas": -- Searching for lib 'hipblas' ... -- Found lib '/opt/rocm-5.6.1/lib/libhipblas.so' -- Searching for a lib in the set "rocblas": -- Searching for lib 'rocblas' ... -- Found lib '/opt/rocm-5.6.1/lib/librocblas.so' -- Searching for a lib in the set "hipsparse": -- Searching for lib 'hipsparse' ... -- Found lib '/opt/rocm-5.6.1/lib/libhipsparse.so' -- Searching for a lib in the set "rocsparse": -- Searching for lib 'rocsparse' ... -- Found lib '/opt/rocm-5.6.1/lib/librocsparse.so' -- Searching for a lib in the set "amd_comgr": -- Searching for lib 'amd_comgr' ... -- Found lib '/opt/rocm-5.6.1/lib/libamd_comgr.so' -- Searching for a lib in the set "hsa-runtime64": -- Searching for lib 'hsa-runtime64' ... -- Found lib '/opt/rocm-5.6.1/lib/libhsa-runtime64.so' -- Searching for a lib in the set "amdhip64": -- Searching for lib 'amdhip64' ... -- Found lib '/opt/rocm-5.6.1/lib/libamdhip64.so' -- TPL_DLlib_LIBRARIES='/usr/lib64/libdl.so;/opt/rocm-5.6.1/lib/libhipsolver.so;/opt/rocm-5.6.1/lib/librocsolver.so;/opt/rocm-5.6.1/lib/libhipblas.so;/opt/rocm-5.6.1/lib/librocblas.so;/opt/rocm-5.6.1/lib/libhipsparse.so;/opt/rocm-5.6.1/lib/librocsparse.so;/opt/rocm-5.6.1/lib/libamd_comgr.so;/opt/rocm-5.6.1/lib/libhsa-runtime64.so;/opt/rocm-5.6.1/lib/libamdhip64.so' Setting up testing support ... -- CTEST_DROP_METHOD='http' -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' Configuring individual enabled Trilinos packages ... Processing enabled top-level package: Kokkos (Libs) -- Setting default Kokkos CXX standard to 17 -- Kokkos version: 4.1.00 -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- The project name is: Trilinos -- Using internal gtest for testing -- Compiler Version: 16.0.0 -- Using -std=c++17 for C++17 standard as feature -- Built-in Execution Spaces: -- Device Parallel: Kokkos::HIP -- Host Parallel: NoTypeDefined -- Host Serial: SERIAL -- -- Architectures: -- VEGA90A -- Using internal desul_atomics copy -- Kokkos Devices: SERIAL;HIP, Kokkos Backends: SERIAL;HIP Processing enabled top-level package: KokkosKernels (Libs) -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- The project name is: Trilinos -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' CMake Warning at packages/kokkos-kernels/cmake/kokkoskernels_eti_devices.cmake:94 (MESSAGE): The CMake option CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS is either undefined or OFF. Please set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL=ON when building with HIP and complex double enabled. Call Stack (most recent call first): packages/kokkos-kernels/CMakeLists.txt:194 (INCLUDE) Overriding KOKKOSKERNELS_ENABLE_TPL_BLAS=OFF with TPL_ENABLE_BLAS=ON Overriding KOKKOSKERNELS_ENABLE_TPL_LAPACK=OFF with TPL_ENABLE_LAPACK=ON Overriding KOKKOSKERNELS_ENABLE_TPL_ROCBLAS=OFF with TPL_ENABLE_ROCBLAS=ON Overriding KOKKOSKERNELS_ENABLE_TPL_ROCSPARSE=OFF with TPL_ENABLE_ROCSPARSE=ON -- Performing Test KK_BLAS_RESULT_AS_POINTER_ARG -- Performing Test KK_BLAS_RESULT_AS_POINTER_ARG - Success ================================ Kokkos Kernels version: 4.1.0 ================================ Kokkos Kernels ETI Types Devices: ; Scalars: double;complex Ordinals: int Offsets: size_t Layouts: LayoutLeft Kokkos Kernels components COMMON: ON BATCHED: ON BLAS: ON GRAPH: ON SPARSE: ON ODE: ON Kokkos Kernels TPLs BLAS: LAPACK: ROCBLAS: ROCSPARSE: ================================ -- Creating ETI files for Batched_Gemm_nt_nt_bll -- Creating ETI files for Batched_Gemm_nt_t_bll -- Creating ETI files for Batched_Gemm_t_nt_bll -- Creating ETI files for Batched_Gemm_t_t_bll -- Creating ETI files for Batched_Gemm_nt_nt_blr -- Creating ETI files for Batched_Gemm_nt_t_blr -- Creating ETI files for Batched_Gemm_t_nt_blr -- Creating ETI files for Batched_Gemm_t_t_blr -- Creating ETI files for Blas1_abs -- Creating ETI files for Blas1_abs_mv -- Creating ETI files for Blas1_scal -- Creating ETI files for Blas1_scal_mv -- Creating ETI files for Blas1_dot -- Creating ETI files for Blas1_dot_mv -- Creating ETI files for Blas_gesv -- Creating ETI files for Blas1_axpby -- Creating ETI files for Blas1_axpby_mv -- Creating ETI files for Blas1_update -- Creating ETI files for Blas1_update_mv -- Creating ETI files for Blas1_sum -- Creating ETI files for Blas1_sum_mv -- Creating ETI files for Blas1_nrm1 -- Creating ETI files for Blas1_nrm1_mv -- Creating ETI files for Blas1_nrm2w -- Creating ETI files for Blas1_nrm2w_mv -- Creating ETI files for Blas1_nrminf -- Creating ETI files for Blas1_nrminf_mv -- Creating ETI files for Blas1_iamax -- Creating ETI files for Blas1_iamax_mv -- Creating ETI files for Blas1_nrm2 -- Creating ETI files for Blas1_nrm2_mv -- Creating ETI files for Blas1_mult -- Creating ETI files for Blas1_mult_mv -- Creating ETI files for Blas1_reciprocal -- Creating ETI files for Blas1_reciprocal_mv -- Creating ETI files for Blas1_rot -- Creating ETI files for Blas1_rotg -- Creating ETI files for Blas1_rotm -- Creating ETI files for Blas1_rotmg -- Creating ETI files for Blas1_swap -- Creating ETI files for Blas2_gemv -- Creating ETI files for Blas2_ger -- Creating ETI files for Blas3_gemm -- Creating ETI files for Blas3_trsm -- Creating ETI files for Blas3_trmm -- Creating ETI files for Blas_trtri -- Creating ETI files for Graph_color_d1 -- Creating ETI files for Sparse_sptrsv_solve -- Creating ETI files for Sparse_spmv_struct -- Creating ETI files for Sparse_spmv_mv_struct -- Creating ETI files for Sparse_spmv_bsrmatrix -- Creating ETI files for Sparse_spmv_mv_bsrmatrix -- Creating ETI files for Sparse_spmv -- Creating ETI files for Sparse_spmv_mv -- Creating ETI files for Sparse_spgemm_symbolic -- Creating ETI files for Sparse_spgemm_numeric -- Creating ETI files for Sparse_spgemm_noreuse -- Creating ETI files for Sparse_bspgemm_numeric -- Creating ETI files for Sparse_spgemm_jacobi -- Creating ETI files for Sparse_spadd_symbolic -- Creating ETI files for Sparse_spadd_numeric -- Creating ETI files for Sparse_spiluk_symbolic -- Creating ETI files for Sparse_spiluk_numeric -- Creating ETI files for Sparse_par_ilut_symbolic -- Creating ETI files for Sparse_par_ilut_numeric -- Creating ETI files for Sparse_gmres -- Creating ETI files for Sparse_sptrsv_symbolic -- Creating ETI files for Sparse_trsv -- Creating ETI files for Sparse_gauss_seidel_symbolic -- Creating ETI files for Sparse_gauss_seidel_numeric -- Creating ETI files for Sparse_gauss_seidel_apply No ETI support requested by packages. Generating dummy makefiles in each directory to call Ninja ... Set up for creating a distribution ... Finished configuring Trilinos! -- If publishing results using Trilinos, please cite us: https://trilinos.github.io/cite.html -- Configuring done (16.8s) -- Generating done (0.0s) CMake Warning: Manually-specified variables were not used by the project: BinUtils_INCLUDE_DIRS BinUtils_LIBRARY_DIRS -- Build files have been written to: /tmp/jjellio/build/amd-5.6.1_prgenv-amd_rocm-5.6.1_mpich-8.1.27_pure-amd_hip_amd-NOinlall-NOfunc-gfx90a_serial_atdm_complex_opt-g_cxx17_libonly/trilinos ```
jjellio commented 10 months ago

Worth pointing out: In this configure script if you don't explicitly enable Kokkos, it enables Kokkos, but does not respect the Kokkos_ENABLE_HIP option (some bug in the Tribits/Cmake mojo). Just a note from when I cut my config down as much as possible (thinking I could get away with Kokkos_ENABLE_HIP=ON and -DTrilinos_ENABLE_KokkosKernels=ON (plus the other stuff)

lucbv commented 10 months ago

Thanks, I'll go have a look and try to reproduce, I'll let you know if I run into some difficulties. Hopefully this should not be too bad to fix!

lucbv commented 10 months ago

@cwpearson looking at the error above, the code in Trilinos and the code in Kokkos Kernels, it seems that this error is due to a bug in the TPL layer for rocsparse that was fixed in Kokkos Kernels but not in Trilinos. Basically going from typename AMatrix::block_layout to typename AMatrix::block_layout_type in KokkosSparse_spmv_bsrmatrix_tpl_spec_decl.hpp:917 should get things to compile right?

cwpearson commented 10 months ago

@jjellio I think we fixed this in https://github.com/trilinos/Trilinos/pull/12330, could you try again and report back? Thanks

jjellio commented 10 months ago

Will do

lucbv commented 9 months ago

ping?