Kokkos + KokkosKernels Promotion To 4.5.0

ndellingwood commented 1 week ago

@trilinos/kokkos @trilinos/kokkos-kernels

Motivation

Snapshot Kokkos Ecosystem 4.5.0 to Trilinos

Related Issues

Possibly blocked by nvcc compiler issues with some intrepid 2 tests, see e.g. https://github.com/trilinos/Trilinos/pull/13571#issuecomment-2463732988 and https://trilinos-cdash.sandia.gov/viewBuildError.php?buildid=1959451

Stakeholder Feedback

Testing

Various nightly integration testing configurations

trilinos-autotester commented 1 week ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 1 week ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 746
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc

Build Num: 796
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 797
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_clang

Build Num: 795
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4754
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda

Build Num: 794
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_intel

Build Num: 715
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda-uvm

Build Num: 794
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 1 week ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 746
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc

Build Num: 796
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 797
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_clang

Build Num: 795
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4754
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda

Build Num: 794
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_intel

Build Num: 715
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda-uvm

Build Num: 794
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	38207da70d79bef1efddbc19c875829fd517a4f4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 1 week ago

Build failures with cuda/11.4.2 due to nvcc errors

FAILED: packages/intrepid2/unit-test/Discretization/Basis/HGRAD_PYR_C1_FEM/CMakeFiles/Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD.dir/test_02_CUDA_DFAD_DFAD.cpp.o 
nvcc error   : 'ptxas' died due to signal 11 (Invalid memory reference)
nvcc error   : 'ptxas' core dumped
FAILED: packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TET_Cn_FEM/CMakeFiles/Intrepid2_unit-test_Discretization_Basis_HGRAD_TET_Cn_FEM_test_02_CUDA_DFAD_DFAD.dir/test_02_CUDA_DFAD_DFAD.cpp.o 
nvcc error   : 'ptxas' died due to signal 11 (Invalid memory reference)
nvcc error   : 'ptxas' core dumped

https://trilinos-cdash.sandia.gov/viewBuildError.php?buildid=1961137

ndellingwood commented 1 week ago

@trilinos/framework @sebrowne is the AT1 PR_cuda job running in a container or directly on machine?

achauphan commented 1 week ago

@ndellingwood AT1 PR_cuda is running on the machine using SEMS modules and not utilizing a container.

ndellingwood commented 1 week ago

I reproduced on the cee hpws

For this nvcc fail FAILED: packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TET_Cn_FEM/CMakeFiles/Intrepid2_unit-test_Discretization_Basis_HGRAD_TET_Cn_FEM_test_02_CUDA_DFAD_DFAD.dir/test_02_CUDA_DFAD_DFAD.cpp.o,

if I remove the outer loop here and replace with a dummy variable ordinal_type i=0; (this breaks correctness of course, but I'm trying to get a sense of where the compiler is struggling), then the code will compile without issue. I'm not sure yet how to address, but have honed in on a spot to work from

ndellingwood commented 1 week ago

Removing any of the loops (and inserting an iteration variable as replacement) in https://github.com/trilinos/Trilinos/blob/develop/packages/intrepid2/src/Discretization/Basis/Intrepid2_HGRAD_TET_Cn_FEMDef.hpp#L74-L80 allows me to compile the test without hitting the nvcc compiler fault e.g. nvcc error : 'ptxas' died due to signal 11 (Invalid memory reference)

A couple added notes:

This is occurring with cuda/11.4.2, reproduced on a cee hpws machine
This is not occuring with cuda/11.2.2 on Power9+Volta70 (weaver)

ndellingwood commented 1 week ago

Some other hacking experiments I tried:

if I leave the loops as is, but replace the addition assignment operator output.access(i,j,k) += vinv(l,i)*phis.access(l,j,k); with assignment (yes, breaks correctness of the code) output.access(i,j,k) = vinv(l,i)*phis.access(l,j,k); then the test compiles without issue
replacing the addition assignment operator with assignment and expanding the expression i.e. output.access(i,j,k) = output.access(i,j,k) + vinv(l,i)*phis.access(l,j,k); still triggers the nvcc segfault

ndellingwood commented 1 week ago

This type of change does not prevent the nvcc issue:

          for (ordinal_type l=0;l<card;++l) {
            auto tmp1 = output.access(i,j,k);
            auto tmp2 = vinv(l,i)*phis.access(l,j,k);
            output.access(i,j,k) = tmp1 + tmp2;
          }

ndellingwood commented 1 week ago

Interchanging a loop order 690972e allowed me to get the code in Intrepid2_HGRAD_TET_Cn_FEMDef.hpp and the associated test02 compiling in my reproducer on cee hpws node with sems-cuda/11.4.2 (I did not reproduce the issue in the HGRAD_PYR_C1_FEM test02 test). I pushed the change to see how it looks in another pass of the AT

trilinos-autotester commented 1 week ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 1 week ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 775
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc

Build Num: 825
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 826
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_clang

Build Num: 824
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4783
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda

Build Num: 823
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_intel

Build Num: 744
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda-uvm

Build Num: 823
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

ndellingwood commented 1 week ago

AT2 cuda-uvm test had these failures:

      5 - Kokkos_CoreUnitTest_Cuda1 (Failed)
      8 - Kokkos_CoreUnitTest_CudaTimingBased (Failed)
     13 - Kokkos_CoreUnitTest_Default (Failed)

These should be resolvable with the RUN_SERIAL property, @trilinos/framework @sebrowne @achauphan what is the process, and where are configurations for the AT2 jobs located, in order to mark these tests RUN_SERIAL?

trilinos-autotester commented 1 week ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 775
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc

Build Num: 825
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 826
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_clang

Build Num: 824
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4783
Status: FAILED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda

Build Num: 823
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_intel

Build Num: 744
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

Build Information

Test Name: PR_cuda-uvm

Build Num: 823
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	690972edc2416e04b32d107ebb2fc38864471bd4
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	813f04eff17fc7b47ced561429f09e86c8c2bb10

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 6 days ago

The cuda-11.4.2 job failed again trying to compile /scratch/trilinos/workspace/PR_cuda/pull_request_test/packages/intrepid2/unit-test/Discretization/Basis/HGRAD_PYR_C1_FEM/test_02_CUDA_DFAD_DFAD.cpp,

nvcc error   : 'ptxas' died due to signal 11 (Invalid memory reference)
nvcc error   : 'ptxas' core dumped

https://trilinos-cdash.sandia.gov/viewBuildError.php?buildid=1973768

The workaround in 690972e resolved packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TET_Cn_FEM/CMakeFiles/Intrepid2_unit-test_Discretization_Basis_HGRAD_TET_Cn_FEM_test_02_CUDA_DFAD_DFAD.dir/test_02_CUDA_DFAD_DFAD.cpp.o

ndellingwood commented 6 days ago

I reproduced the cuda-11.4.2 failure /scratch/trilinos/workspace/PR_cuda/pull_request_test/packages/intrepid2/unit-test/Discretization/Basis/HGRAD_PYR_C1_FEM/test_02_CUDA_DFAD_DFAD.cpp, with a clean rebuild of this branch on a cee hpws machine building with a lot of parallelism, make -j48

However, when I rebuild the test target alone (so that I could begin to triage), the compilation is successful

[ndellin@hpws0594 SemsCuda1142OpenMPI416]$ make Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD
ninja -C .  Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD
ninja: Entering directory `.'
[1/1] Linking CXX executable packages/intrepid2/unit-test/Discretization/Basis/HGRAD_PYR_C1_FEM/Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD.exe

I'm not sure how to proceed, I can't investigate if the issue goes away when recompiling. Does this indicate that this particular test struggles due to the high level of build parallelism? @trilinos/framework @sebrowne @mperego

ndellingwood commented 6 days ago

Rebuilding with reduced parallelism (make -j16) I did not run into the HGRAD_PYR_C1_FEM related nvcc issues, but some of the Projection tests failed:

FAILED: packages/intrepid2/unit-test/Projection/CMakeFiles/Intrepid2_unit-test_Projection_test_convergence_TRI_CUDA_DOUBLE.dir/test_convergence_TRI_CUDA_DOUBLE.cpp.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
...
FAILED: packages/intrepid2/unit-test/Projection/CMakeFiles/Intrepid2_unit-test_Projection_test_convergence_PYR_CUDA_DOUBLE.dir/test_convergence_PYR_CUDA_DOUBLE.cpp.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)

But if I isolate and manually compile the cpp file that failed, I don't hit the nvcc error above, e.g. this worked when I manually ran this line:

[ndellin@hpws0594 SemsCuda1142OpenMPI416]$ mpic++ -DKOKKOS_DEPENDENCE -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416 -I/gpfs/ndellin/Trilinos -I/gpfs/ndellin/Trilinos/packages/intrepidunit-test/Projection -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/intrepid2/src -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Cell -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Discretization/Basis -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Discretization/FunctionSpaceTools -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Discretization/Integration -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Orientation -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Projection -I/gpfs/ndellin/Trilinos/packages/intrepid2/src/Shared -I/gpfs/ndellin/Trilinos/packages/intrepid2/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/teuchos/core/src -I/gpfs/ndellin/Trilinos/packages/teuchos/core/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos/core/src -I/gpfs/ndellin/Trilinos/packages/kokkos/core/src -I/gpfs/ndellin/Trilinos/packages/kokkos/tpls/desul/include -I/gpfs/ndellin/Trilinos/packages/kokkos/tpls/mdspan/include -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos/containers/src -I/gpfs/ndellin/Trilinos/packages/kokkos/containers/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos/algorithms/src -I/gpfs/ndellin/Trilinos/packages/kokkos/algorithms/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos/simd/src -I/gpfs/ndellin/Trilinos/packages/kokkos/simd/src -I/gpfs/ndellin/Trilinos/packages/teuchos/numerics/src -I/gpfs/ndellin/Trilinos/packages/teuchos/comm/src -I/gpfs/ndellin/Trilinos/packages/teuchos/parameterlist/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/teuchos/parameterlist/src -I/gpfs/ndellin/Trilinos/packages/teuchos/parser/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/shards/src -I/gpfs/ndellin/Trilinos/packages/shards/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/sacado/src -I/gpfs/ndellin/Trilinos/packages/sacado/src -I/gpfs/ndellin/Trilinos/packages/sacado/src/new_design -I/gpfs/ndellin/Trilinos/packages/sacado/src/template -I/gpfs/ndellin/Trilinos/packages/sacado/src/parameter -I/gpfs/ndellin/Trilinos/packages/sacado/src/mpl -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/teuchos/kokkoscomm/src -I/gpfs/ndellin/Trilinos/packages/teuchos/kokkoscomm/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/teuchos/kokkoscompat/src -I/gpfs/ndellin/Trilinos/packages/teuchos/kokkoscompat/src -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/batched -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/blas -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/blas -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/lapack -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/lapack -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/graph -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/graph -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/ode -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/common/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/common/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/common/unit_test -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/batched/eti -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/dense/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/dense/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/dense/unit_test -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/sparse/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/sparse/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/batched/sparse/unit_test -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/blas/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/blas/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/blas/eti -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/blas/eti -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/blas/tpls -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/lapack/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/lapack/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/lapack/eti -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/lapack/eti -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/lapack/tpls -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/graph/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/graph/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/graph/eti -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/graph/eti -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/sparse/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/sparse/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/sparse/eti -I/gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/kokkos-kernels/sparse/eti -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/sparse/tpls -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/ode/src -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/ode/impl -I/gpfs/ndellin/Trilinos/packages/kokkos-kernels/ode/unit_test -isystem /projects/sems/install/rhel8-x86_64/sems/tpl/cuda/11.4.2/gcc/10.1.0/base/aye3tix/include -isystem /projects/sems/install/rhel8-x86_64/sems/tpl/boost/1.74.0/gcc/10.1.0/base/bcbribv/include -isystem /projects/sems/install/rhel8-x86_64/sems/tpl/metis/5.1.0/gcc/10.1.0/base/pts2hxl/include -pedantic -Wall -Wno-long-long -Wwrite-strings   -fPIC -Wall -Warray-bounds -Wchar-subscripts -Wcomment -Wenum-compare -Wformat -Wuninitialized -Wmaybe-uninitialized -Wmain -Wnarrowing -Wnonnull -Wreorder -Wreturn-type -Wsequence-point -Wtrigraphs -Wunused-function -Wunused-but-set-variable -Wwrite-strings -O3 -DNDEBUG -extended-lambda -Wext-lambda-captures-this -arch=sm_70 -std=c++17 -MD -MT packages/intrepid2/unit-test/Projection/CMakeFiles/Intrepid2_unit-test_Projection_test_convergence_TRI_CUDA_DOUBLE.dir/test_convergence_TRI_CUDA_DOUBLE.cpp.o -MF packages/intrepid2/unit-test/Projection/CMakeFiles/Intrepid2_unit-test_Projection_test_convergence_TRI_CUDA_DOUBLE.dir/test_convergence_TRI_CUDA_DOUBLE.cpp.o.d -o packages/intrepid2/unit-test/Projection/CMakeFiles/Intrepid2_unit-test_Projection_test_convergence_TRI_CUDA_DOUBLE.dir/test_convergence_TRI_CUDA_DOUBLE.cpp.o -c /gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416/packages/intrepid2/unit-test/Projection/test_convergence_TRI_CUDA_DOUBLE.cpp

trilinos-autotester commented 1 day ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 804
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 854
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 855
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 853
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4812
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 852
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 773
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 852
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 804
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 854
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 855
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 853
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4812
Status: FAILED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 852
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 773
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 852
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	5458ac8939f6b7ce2b2e1af12be5e1a1ed07dab3
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 1 day ago

In the most recent retest, Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD_MPI_1 was disabled which allowed the cuda-11.4.2 job to complete compilation 👍

The PR_cuda job had an Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 test failure in the cuda-11.4.2 job, the first couple lines of relevant output:
```
order: 4, ic: 0, i: 3, j: 8, curls A: [-4.50039, 21.5114], curls B: [-4.50039, 21.5114], |diff|: 8.88178e-16, tol: 2.22045e-16
order: 4, ic: 0, i: 4, j: 8, curls A: [-1.59564e-15, -1.25463], curls B: [-2.65035e-15, -1.25463], |diff|: 1.05471e-15, tol: 2.22045e-16
```
@mperego is this a case where the tolerance needs to be slightly loosened? If not, can you help advise?
The second failure in the PR_gcc-openmpi-openmp job in the Belos_bl_cg_real_hb_1_MPI_4 job due to apparent out-of-memory issue; this did not occur in the previous runs of this PR, and the most recent commit only disabled an intrepid2 test with cuda-11.4.2 jobs, @sebrowne @trilinos/framework can you please help advise?
```
Real format () in H/B file not supported.
Insufficient memory for ThisElement.--------------------------------------------------------------------------
```

The Trilinos_PR_python3 build had 3 failures One compiler error with a -Werror in Sacado's copy of gtest, but that was unchanged with this PR, @trilinos/framework @sebrowne can you help advise why that error occurred? Error building packages/sacado/test/GTestSuite/googletest/googletest/CMakeFiles/sacado-gtest.dir/src/gtest-all.cc.o

In file included from ../Trilinos/packages/sacado/test/GTestSuite/googletest/googletest[gtest-all.cc:42](https://github.com/trilinos/Trilinos//blob/f99ccd3746ed83249102f26b036e9c0ce674be9c/gtest-all.cc#L42):
../Trilinos/packages/sacado/test/GTestSuite/googletest/googletest/src/gtest-death-test.cc: In function 'bool testing::internal::StackGrowsDown()':
../Trilinos/packages/sacado/test/GTestSuite/googletest/googletest[gtest-death-test.cc:1306](https://github.com/trilinos/Trilinos//blob/f99ccd3746ed83249102f26b036e9c0ce674be9c/gtest-death-test.cc#L1306):24: error: 'dummy' may be used uninitialized [-Werror=maybe-uninitialized]
1306 |   StackLowerThanAddress(&dummy, &result);
  |   ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[CTest: warning matched] ../Trilinos/packages/sacado/test/GTestSuite/googletest/googletest[gtest-death-test.cc:1295](https://github.com/trilinos/Trilinos//blob/f99ccd3746ed83249102f26b036e9c0ce674be9c/gtest-death-test.cc#L1295):13: note: by argument 1 of type 'const void*' to 'void testing::internal::StackLowerThanAddress(const void*, bool*)' declared here
1295 | static void StackLowerThanAddress(const void* ptr, bool* result) {
  |             ^~~~~~~~~~~~~~~~~~~~~
[CTest: warning matched] ../Trilinos/packages/sacado/test/GTestSuite/googletest/googletest[gtest-death-test.cc:1304](https://github.com/trilinos/Trilinos//blob/f99ccd3746ed83249102f26b036e9c0ce674be9c/gtest-death-test.cc#L1304):7: note: 'dummy' declared here
1304 |   int dummy;
  |       ^~~~~
[CTest: warning matched] cc1plus: all warnings being treated as errors

2 unit test failures

TrilinosInstallTests_reduced_tarball failed, likely related to #13615
A SEACASIoss_pamgen_exodus_io_info test that did not occur in previous runs and unrelated to changes in this PR, looks like a configuration issues, @trilinos/framework please help advise
```
...
ERROR: The database type 'exodus' is not supported.
```

Supported database types: catalyst_cgns catalyst_exodus gen_struc generated heartbeat null pamgen textmesh

mperego commented 1 day ago

@ndellingwood Loosening the tolerance will fix it, but I'm a bit surprised we need to do that. Feel free to disable that test. Once this is merged I'll find some time to work on this.

sebrowne commented 1 day ago

Ignore all of the failures on the aue python3 line, they are spurious (and won't block the PR moving forwards, I fixed the non-blocking build capability right after that run began).

The Belos one is a bit suspicious, I checked the monitoring on the machine that ran that set of tests and its memory looked just fine. Also don't see it failing anywhere in recent history. Perhaps a RETEST is appropriate for that one? Will have to RETEST anyway to clear the aue/python3 line....

trilinos-autotester commented 1 day ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 812
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 862
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 863
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 861
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4820
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 860
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 781
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 860
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 812
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 862
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 863
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 861
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4820
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 860
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 781
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 860
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	756cdd60dc1d2b2fa116a13b21548a43975b8df8
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 1 day ago

My attempt to disable in the cuda-11.4.2 job Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 didn't work and the failure triggered again, I'll push another update in case I added the disable in the wrong location

Separately in the PR_clang build, a spurious failure in TeuchosCore_RCP_PerformanceTests_basic_MPI_1

trilinos-autotester commented 1 day ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 814
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 864
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 865
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 863
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4822
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 862
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 783
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 862
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 1 day ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 814
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 864
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 865
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 863
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4822
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 862
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 783
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 862
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	e0cb35641454b98eb264684e05e0e90431b88166
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 1 day ago

In the PR_cuda build the Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 test failed again, though I thought the changes in 756cdd6 should disable it (I added the change e0cb356 though that should only be on ATS2, @trilinos/framework @mperego do you see a mistake I made in disabling the test?

sebrowne commented 20 hours ago

@ndellingwood I don't think disabling the test disables the build of the test (at least that's the behavior I'm seeing). I'm testing a change in https://github.com/trilinos/Trilinos/pull/13439 that automatically disables those tests for CUDA <12.

ndellingwood commented 19 hours ago

@sebrowne applying the_DISABLE property to the Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD_MPI_1 test worked to remove it from the build (and allow the PR_cuda compilation to complete), but for some reason when applied to the Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 test it is still building and running, not clear to me yet why there is a discrepancy (naming convention of the tests is very similar)

ndellingwood commented 15 hours ago

I did some testing in my reproducer build to disable the Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 unit test:

I removed all the .exe files from packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TRI_Cn_FEM
I deleted my CMakeCache.txt and CMakeFiles
I added the -DIntrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1_DISABLE=ON option to my configure line and reconfigured

I rebuilt the HGRAD_TRI_Cn_FEM unit tests cd packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TRI_Cn_FEM; make -j16 ; the Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE.exe test is present (DISABLE did not prevent compilation, which was suggested)

[ndellin@hpws0594 HGRAD_TRI_Cn_FEM]$ ls
CMakeFiles                                                                                  Makefile
cmake_install.cmake                                                                         out.make
CTestTestfile.cmake                                                                         test_01_CUDA_DFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_DFAD_DFAD.exe        test_01_CUDA_DOUBLE_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_DOUBLE_DOUBLE.exe    test_01_CUDA_SFAD_SFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SFAD_SFAD.exe        test_01_CUDA_SLFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SLFAD_DFAD.exe       test_01_CUDA_SLFAD_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SLFAD_DOUBLE.exe     test_01_Serial_DFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_DFAD_DFAD.exe      test_01_Serial_DOUBLE_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_DOUBLE_DOUBLE.exe  test_01_Serial_SFAD_SFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SFAD_SFAD.exe      test_01_Serial_SLFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SLFAD_DFAD.exe     test_01_Serial_SLFAD_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SLFAD_DOUBLE.exe   test_02_CUDA_DFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DFAD_DFAD.exe        test_02_CUDA_DOUBLE_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE.exe    test_02_Serial_DFAD_DFAD.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_Serial_DFAD_DFAD.exe      test_02_Serial_DOUBLE_DOUBLE.cpp
Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_Serial_DOUBLE_DOUBLE.exe

The test was removed from the list of tests to execute via ctest:


[ndellin@hpws0594 HGRAD_TRI_Cn_FEM]$ ctest -N .
Test project /gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416-ninja-8/packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TRI_Cn_FEM
Test  #1: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_DOUBLE_DOUBLE_MPI_1
Test  #2: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SFAD_SFAD_MPI_1
Test  #3: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_DFAD_DFAD_MPI_1
Test  #4: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SLFAD_DFAD_MPI_1
Test  #5: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_Serial_SLFAD_DOUBLE_MPI_1
Test  #6: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_DOUBLE_DOUBLE_MPI_1
Test  #7: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SFAD_SFAD_MPI_1
Test  #8: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_DFAD_DFAD_MPI_1
Test  #9: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SLFAD_DFAD_MPI_1
Test #10: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_01_CUDA_SLFAD_DOUBLE_MPI_1
Test #11: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_Serial_DOUBLE_DOUBLE_MPI_1
Test #12: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_Serial_DFAD_DFAD_MPI_1
Test #13: Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DFAD_DFAD_MPI_1

Total Tests: 13 [ndellin@hpws0594 HGRAD_TRI_Cn_FEM]$ ctest -R Intrepid2_unit-test_Discretization_Basis_HGRAD_TRI_Cn_FEM_test_02_CUDA_DOUBLE_DOUBLE_MPI_1 Test project /gpfs/ndellin/Trilinos/Build/SemsCuda1142OpenMPI416-ninja-8/packages/intrepid2/unit-test/Discretization/Basis/HGRAD_TRI_Cn_FEM No tests were found!!!



@sebrowne is it possible to blow away the CMakeCache before a retest for the PR_cuda job? I'm curious if the option I added to the `config-specs.ini` was not enabled since the previous build has already successfully built the test?

trilinos-autotester commented 14 hours ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 14 hours ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 818
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 868
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 869
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 867
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4826
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 866
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 787
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 866
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

sebrowne commented 14 hours ago

@ndellingwood we always build in a fresh directory for the AT builds. My solution to this is here: https://github.com/trilinos/Trilinos/pull/13439/commits/98c0af1c360434fc7af609806164851cdf6d0a9f

I read the TriBITS source code and I'm about 85% sure that there is no way to conditionally disable building a specific test in TriBITS (which is fine, that honestly makes sense), so I added logic directly in the build system. I think this may even be more preferable for tests that we have no intention of repairing with specific toolchains, because now nobody needs to set the DISABLE variables to prevent compiler errors.

trilinos-autotester commented 11 hours ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 818
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 868
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 869
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 867
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4826
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 866
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 787
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 866
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	429b1cd83e68a7244eaed2f3a0a837ab0ea66945
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

CDash Test Results for PR# 13589.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 11 hours ago

The Error building packages/intrepid2/unit-test/Discretization/Basis/HGRAD_PYR_C1_FEM/CMakeFiles/Intrepid2_unit-test_Discretization_Basis_HGRAD_PYR_C1_FEM_test_02_CUDA_DFAD_DFAD.dir/test_02_CUDA_DFAD_DFAD.cpp.o is back in the most recent rebuild :(

I made changes directly in the build system to disable the problematic tests here 3df02b0, as suggested in https://github.com/trilinos/Trilinos/commit/98c0af1c360434fc7af609806164851cdf6d0a9f

trilinos-autotester commented 10 hours ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 10 hours ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 820
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 870
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 871
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 869
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4828
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 868
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 789
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 868
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 9 hours ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 820
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc

Build Num: 870
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 871
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_clang

Build Num: 869
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4828
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda

Build Num: 868
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_intel

Build Num: 789
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

Build Information

Test Name: PR_cuda-uvm

Build Num: 868
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS	pkg: Kokkos;pkg: KokkosKernels
PULLREQUESTNUM	13589
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	3df02b06be946be50ce2567e641acefd24d0291c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	c87e9bdd97c69766e398ee5a0b687d67c93d3466

CDash Test Results for PR# 13589.

trilinos-autotester commented 9 hours ago

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

trilinos-autotester commented 9 hours ago

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

trilinos / Trilinos