MueLu: MueLu_CreateOperatorTpetra_MPI_1 unit test failure in Hip builds

ndellingwood commented 6 months ago

Bug Report

@trilinos/muelu

Description

Hip builds (rocm/5.6.1 on MI210) have the MueLu_CreateOperatorTpetra_MPI_1 failing with diffs between the output and gold files:

23:36:56 3/4 Test #689: MueLu_CreateOperatorTpetra_MPI_1 ............................................***Failed  Required regular expression not found. Regex=[End Result: TEST PASSED
23:36:56 ]  4.60 sec
23:36:56 ==========================================================================================================================
23:36:56 Xpetra::Parameters
23:36:56  Linear algebra library: Tpetra
23:36:56 Galeri::Xpetra::Parameters<long long>
23:36:56  Matrix type: Laplace2D
23:36:56  Problem size: 10000 (100x100)
23:36:56 Processor subdomains in x direction: 1
23:36:56 Processor subdomains in y direction: 1
23:36:56 Processor subdomains in z direction: -1
23:36:56 ========================================================
23:36:56 Testing folder "kokkos/"
23:36:56 ==========================================================================================================================
23:36:56 ===================================== Solve 1: Standard =====================================
23:36:56 --- kokkos/Output/operator_solve_1_np1_tpetra.gold_filtered    2024-05-01 23:35:00.404941557 -0600
23:36:56 +++ kokkos/Output/operator_solve_1_np1_tpetra.out_filtered 2024-05-01 23:35:00.411398308 -0600
23:36:56 @@ -129,6 +129,6 @@
23:36:56  
23:36:56  Smoother (level 2) both : "Ifpack2::Relaxation": {Initialized: true, Computed: true, Type: Symmetric Gauss-Seidel, sweeps: 1, damping factor: 1, Global matrix dimensions: [192, 192], Global nnz: 1682}
23:36:56  
23:36:56 -Smoother (level 3) pre  : <Direct> solver interface
23:36:56 +Smoother (level 3) pre  : KLU2 solver interface
23:36:56  Smoother (level 3) post : no smoother
...

The first diff that pops out is:

23:36:56 -Smoother (level 3) pre  : <Direct> solver interface
23:36:56 +Smoother (level 3) pre  : KLU2 solver interface

Steps to Reproduce

SHA1: 8fb65ea645b7c02d49f17c2b44bb45b253f04d13

Configure script: caraway MI210 queue


module load python rocm/5.6.1 cmake openmpi/4.1.5 openblas/0.3.23 ninja/1.11.1
export OMPI_CXX=$ROCM_PATH/bin/hipcc
export TPETRA_ASSUME_GPU_AWARE_MPI=0

cmake \ -G"Ninja" \ -DCMAKE_INSTALL_PREFIX=$PWD/install \ -DCMAKE_CXX_STANDARD="17" \ -DCMAKE_CXX_COMPILER="which mpicxx" \ -DCMAKE_C_COMPILER="which mpicc" \ -DCMAKE_FORTRAN_COMPILER="which mpifort" \ -DCMAKE_BUILD_TYPE="RELEASE" \ -DBUILD_SHARED_LIBS="OFF" \ \ -DTrilinos_ENABLE_ALL_PACKAGES=OFF \ -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \ -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \ -DTrilinos_ASSERT_MISSING_PACKAGES=OFF \ -DTrilinos_ALLOW_NO_PACKAGES=OFF \ -DTrilinos_ENABLE_OpenMP=OFF \ -DTrilinos_ENABLE_TESTS=ON \ \ -DTrilinos_ENABLE_Amesos2=ON \ -DAmesos2_ENABLE_SuperLU=OFF \ -DAmesos2_ENABLE_KLU2=ON \ -DTrilinos_ENABLE_Belos=ON \ -DTrilinos_ENABLE_Ifpack2=ON \ -DTrilinos_ENABLE_Kokkos=ON \ -DKokkos_ARCH_VEGA90A=ON \ -DKokkos_ENABLE_CUDA=OFF \ -DKokkos_ENABLE_HIP=ON \ -DKokkos_ENABLE_OPENMP=OFF \ -DTrilinos_ENABLE_KokkosKernels=ON \ -DTrilinos_ENABLE_MueLu=ON \ -DTrilinos_ENABLE_Tpetra=ON \ -DTpetra_ENABLE_CUDA=OFF \ -DTpetra_INST_HIP=ON \ -DTpetra_INST_SERIAL=OFF \ -DTpetra_INST_OPENMP=OFF \ -DTpetra_INST_DOUBLE=ON \ -DTrilinos_ENABLE_Gtest=ON \ -DTrilinos_ENABLE_Teuchos=ON \ -DTrilinos_ENABLE_Xpetra=ON \ -DTrilinos_ENABLE_Zoltan2=ON \ -DTrilinos_ENABLE_Panzer=ON \ -DTPL_ENABLE_BLAS=ON \ -D BLAS_LIBRARY_DIRS:FILEPATH="${OPENBLAS_ROOT}/lib" \ -D BLAS_LIBRARY_NAMES:STRING="openblas" \ -DTPL_ENABLE_LAPACK=ON \ -D LAPACK_INCLUDE_DIRS:FILEPATH="${OPENBLAS_ROOT}/include" \ -D LAPACK_LIBRARY_DIRS:FILEPATH="${OPENBLAS_ROOT}/lib" \ -D LAPACK_LIBRARY_NAMES:STRING="openblas" \ -DTPL_ENABLE_Netcdf=OFF \ -DTPL_ENABLE_MPI=ON \ -DMPI_USE_COMPILER_WRAPPERS=ON \ -DMPI_EXEC="mpirun" \ -DMPI_EXEC_NUMPROCS_FLAG="-np" \ -DMPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-bind-to;none" \ \ $TRILINOS_DIR

github-actions[bot] commented 6 months ago

Automatic mention of the @trilinos/muelu team

github-actions[bot] commented 6 months ago

Automatic mention of the @trilinos/muelu team

trilinos / Trilinos