Ifpack2: Block TriDiag test fails with device aware MPI off but HIP enabled

[jjellio]

Bug Report

I've built and run the ifpack2 block tridiag test, but if I disable device aware MPI (at the MPI and Tpetra level), the app will crash.

@vbrunini @kliegeois

# GPU-aware (using 2 procs) - works
MPICH_GPU_SUPPORT_ENABLED=1 \
 TPETRA_ASSUME_GPU_AWARE_MPI=1 \
 flux \
 run \
 -c8 \
 -g1 \
 -n \
 2 \
 ./packages/ifpack2/example/Ifpack2_BlockTriDiagonalSolver.exe \
 --withoutOverlapCommAndComp \
 --matrixType=Laplace3D \
 --blockSize=11 \
 --nx=64 \
 --ny=32 \
 --nz=64 \
 --sublinesPerLine=1 \
 --tol=0 \
 --numIters=10000 \
 --withStackedTimer

#disable GPU-aware MPI - it fails
MPICH_GPU_SUPPORT_ENABLED=0 \
 TPETRA_ASSUME_GPU_AWARE_MPI=0 \
 flux \
 run \
 -c8 \
 -g1 \
 -n \
 2 \
 ./packages/ifpack2/example/Ifpack2_BlockTriDiagonalSolver.exe \
 --withoutOverlapCommAndComp \
 --matrixType=Laplace3D \
 --blockSize=11 \
 --nx=64 \
 --ny=32 \
 --nz=64 \
 --sublinesPerLine=1 \
 --tol=0 \
 --numIters=10000 \
 --withStackedTimer

configure

``` cmake \ "-GNinja" \ "-DCMAKE_BUILD_TYPE:STRING=Release" \ "-DTrilinos_ENABLE_TrilinosBuildStats=OFF" \ "-DTrilinos_ENABLE_BUILD_STATS=OFF" \ "-DTrilinosBuildStats_ENABLE_TESTS=OFF" \ "-DBUILD_SHARED_LIBS:BOOL=OFF" \ "-DTrilinos_ENABLE_Teuchos=ON" \ "-DPanzerMiniEM_ENABLE_EXAMPLES=ON" \ "-DPanzerMiniEM_ENABLE_TESTS=ON" \ "-DTrilinos_ENABLE_Percept=OFF" \ "-DTrilinos_ENABLE_ALL_PACKAGES=ON" \ "-DKokkos_ARCH_ZEN3:BOOL=ON" \ "-DKokkos_ENABLE_SERIAL:BOOL=ON" \ "-DTpetra_INST_SERIAL:BOOL=ON" \ "-DKokkos_ENABLE_OPENMP:BOOL=OFF" \ "-DTpetra_INST_OPENMP:BOOL=OFF" \ "-DTrilinos_ENABLE_OpenMP:BOOL=OFF" \ "-DKokkos_ENABLE_HIP:BOOL=ON" \ "-DTpetra_INST_HIP:BOOL=ON" \ "-DKOKKOSKERNELS_ENABLE_TPL_BLAS:BOOL=ON" \ "-DKOKKOSKERNELS_ENABLE_TPL_LAPACK:BOOL=ON" \ "-DKOKKOSKERNELS_ENABLE_TPL_ROCBLAS:BOOL=ON" \ "-DKOKKOSKERNELS_ENABLE_TPL_ROCSPARSE:BOOL=ON" \ "-DKokkos_ARCH_VEGA90A:BOOL=ON" \ "-DCMAKE_EXE_LINKER_FLAGS=-x none --hip-link " \ "-DCMAKE_SHARED_LINKER_FLAGS=-x none --hip-link " \ "-DFC_FN_UNDERSCORE=UNDER" \ "-DCMAKE_CXX_COMPILER=/opt/cray/pe/craype/2.7.21/bin/CC" \ "-DCMAKE_C_COMPILER=/opt/cray/pe/craype/2.7.21/bin/cc" \ "-DCMAKE_Fortran_COMPILER=/opt/cray/pe/craype/2.7.21/bin/ftn" \ "-DCMAKE_CXX_FLAGS=-x hip -mllvm -amdgpu-early-inline-all=false -mllvm -amdgpu-function-calls=false -g " \ "-DCMAKE_Fortran_FLAGS=" \ "-DCMAKE_C_FLAGS=" \ "-DTrilinos_EXTRA_LINK_FLAGS=" \ .... ``` The standard TPLs enabled ... the compiler definitions are benign. Build how you like. but make sure to use my CXX flags to mimic SPARC

Steps to Reproduce

1. Latest develop
2. Configure script: above (truncated, ping me if you can't reproduce)
3. launch commands above

[github-actions[bot]]

Automatic mention of the @trilinos/ifpack2 team

[github-actions[bot]]

Automatic mention of the @trilinos/ifpack2 team

[kliegeois]

Closing as completed.