Kokkos prerelease test - Githubissues

ndellingwood commented 3 weeks ago

@trilinos/kokkos-kernels @trilinos/kokkos

Motivation

Testing of release candidate branches

Related Issues

Closes put-issue-number-here

Stakeholder Feedback

Testing

AT for additional feedback to nightly integration builds

ndellingwood commented 3 weeks ago

PR for additional testing feedback, @crtrott @lucbv @dalg24 @masterleinad

trilinos-autotester commented 3 weeks ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 3 weeks ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 695
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_gcc

Build Num: 745
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 746
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_clang

Build Num: 744
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4703
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_cuda

Build Num: 743
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_intel

Build Num: 664
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_cuda-uvm

Build Num: 743
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 3 weeks ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 695
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_gcc

Build Num: 745
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 746
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_clang

Build Num: 744
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4703
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_cuda

Build Num: 743
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_intel

Build Num: 664
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

Build Information

Test Name: PR_cuda-uvm

Build Num: 743
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	19661dfa81a22600ecac38e07efd3a2868824e68
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	76e401dbc71e6dcb612f9b16584e0ec67dc9e645

CDash Test Results for PR# 13571.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 3 weeks ago

The rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables job had 36 failures in intrepid2 unit tests

Sample output snips from Intrepid2_unit-test_Discretization_Basis_HCURL_HEX_In_FEM_test_02_CUDA_DFAD_DFAD_MPI_1 :

...
Order: 1: Computing values and curls for 5 cells and 10 points using team-level getValues function
Max Recommended team size: 4, Requested team size: 10
Max Recommended team size: 4, Requested team size: 10
[Intrepid2] Error in file /scratch/trilinos/workspace/PR_cuda/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_HCURL_HEX_In_FEMDef.hpp, line 618
# repeated many times
            Test that evaluated to true: !((subcellDim == -1) && (subcellOrdinal == -1))
# repeated many times
            >>> ERROR: (Intrepid2::Basis_HCURL_HEX_In_FEM::getValues), The capability of selecting subsets of basis functions has not been implemented yet. 
# repeated many times
:0: : block: [0,0,0], thread: [0,0,0] Assertion `[Intrepid2] Abort
` failed.
# similar output repeated many times

The output message pattern is similar from the sample of unit test failures I looked at

@trilinos/intrepid2 @mperego @CamelliaDPG can you help me understand the output messages? In particular, what would trigger the >>> ERROR: (Intrepid2::Basis_HCURL_HEX_In_FEM::getValues), The capability of selecting subsets of basis functions has not been implemented yet message?

mperego commented 2 weeks ago

That's weird. Any reason why the default initialization of subcellDim and subcellOrdinal would not work? https://github.com/trilinos/Trilinos/blob/develop/packages/intrepid2/src/Discretization/Basis/Intrepid2_HCURL_HEX_In_FEM.hpp#L196C1-L203C62

Or maybe is the INTREPID2_TEST_FOR_ABORT macro that is not working?

mperego commented 2 weeks ago

@ndellingwood For context, at the moment we always rely on the default initialization of subcellDim and subcellOrdinal to -1. We added these extra parameters for future capabilities. Maybe it's worth printing out subcellDim and subcellOrdinal.

I'm wondering if this is not a red herring and the issue is somewhere else.

ndellingwood commented 2 weeks ago

@mperego thanks for the info, this is helpful. Are subcellDim and/or subcellOrdinal used for setting a team_size or related variable in the getValues function? If so, the use of -1 may be incompatible with updates to internal handling of negative team size, for example in https://github.com/kokkos/kokkos/pull/7423

mperego commented 2 weeks ago

@mperego thanks for the info, this is helpful. Are subcellDim and/or subcellOrdinal used for setting a team_size or related variable in the getValues function? If so, the use of -1 may be incompatible with updates to internal handling of negative team size, for example in kokkos/kokkos#7423

No, at the moment they are not used at all.

ndellingwood commented 2 weeks ago

@mperego thanks for clarifying (I missed that you mentioned that prior). I'm working on reproducing, hopefully the debugger gives me useful info to share

ndellingwood commented 2 weeks ago

Stacktrace for Intrepid2_unit-test_Discretization_Basis_HCURL_HEX_In_FEM_test_02_CUDA_DFAD_DFAD_MPI_1

#0  0x00007fffa788b94c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fffa783e646 in raise () from /lib64/libc.so.6
#2  0x00007fffa78287f3 in abort () from /lib64/libc.so.6
#3  0x00000000008dd789 in Kokkos::Impl::host_abort(char const*) ()
#4  0x0000000000904ed4 in Kokkos::Impl::cuda_internal_error_abort(cudaError, char const*, char const*, int) ()
#5  0x0000000000900c57 in Kokkos::CudaSpace::impl_deallocate(char const*, void*, unsigned long, unsigned long, Kokkos_Profiling_SpaceHandle) const ()
#6  0x0000000000900cde in Kokkos::CudaSpace::deallocate(char const*, void*, unsigned long, unsigned long) const ()
#7  0x0000000000904b7b in Kokkos::Impl::CudaInternal::resize_team_scratch_space(int, long, bool) ()
#8  0x00000000004dfa76 in Kokkos::Impl::ParallelFor<__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> > >*, Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int>, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>::ParallelFor(__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> > >*, Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&) ()
#9  0x00000000004dff2d in void Kokkos::parallel_for<Kokkos::TeamPolicy<Kokkos::Cuda>, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> > >*, Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int>, void>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> > >*, Kokkos::DynRankView<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&) ()
#10 0x000000000047f477 in int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool) [clone .isra.0] ()
#11 0x0000000000430d16 in main ()

ndellingwood commented 2 weeks ago

Stacktrace for Intrepid2_unit-test_Discretization_Basis_HCURL_HEX_In_FEM_test_02_CUDA_DOUBLE_DOUBLE.exe:

gdb

#0  0x00007fffa788b94c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fffa783e646 in raise () from /lib64/libc.so.6
#2  0x00007fffa78287f3 in abort () from /lib64/libc.so.6
#3  0x0000000000b573f9 in Kokkos::Impl::host_abort (message=<optimized out>) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/impl/Kokkos_Abort.cpp:40
#4  0x0000000000b7eea4 in Kokkos::Impl::cuda_internal_error_abort (e=cudaErrorAssert, name=name@entry=0xc8cf0b "cudaFree(arg_alloc_ptr)", 
    file=file@entry=0xc8cc28 "/ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp", line=line@entry=379)
    at /net/watson.sandia.gov/storage/fast/projects/sems/install/rhel9-x86_64/sems-compilers/gcc/11.4.0/gcc/11.4.1/eptquy7/bin/../lib/gcc/x86_64-pc-linux-gnu/11.4.0/../../../../include/c++/11.4.0/bits/basic_string.h:194
#5  0x0000000000b7ac27 in Kokkos::Impl::cuda_internal_safe_call (file=0xc8cc28 "/ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp", line=379, 
    name=0xc8cf0b "cudaFree(arg_alloc_ptr)", e=<optimized out>) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Error.hpp:61
#6  Kokkos::CudaSpace::impl_deallocate (this=0x7fffffff33f0, arg_label=0xc8c4f2 "[unlabeled]", arg_alloc_ptr=0x7fff8bbdc200, arg_alloc_size=<optimized out>, 
    arg_logical_size=<optimized out>, arg_handle=...) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp:379
#7  0x0000000000b7acae in Kokkos::CudaSpace::deallocate (this=this@entry=0x7fffffff33f0, arg_label=arg_label@entry=0xc8c4f2 "[unlabeled]", arg_alloc_ptr=0x7fff8bbdc200, 
    arg_alloc_size=2800, arg_logical_size=arg_logical_size@entry=0) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp:348
#8  0x0000000000b7acd5 in Kokkos::CudaSpace::deallocate (this=this@entry=0x7fffffff33f0, arg_alloc_ptr=<optimized out>, arg_alloc_size=<optimized out>)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp:343
#9  0x0000000000b7eb4b in Kokkos::Impl::CudaInternal::resize_team_scratch_space (this=this@entry=0x1455b80 <Kokkos::Impl::CudaInternal::singleton()::self>, 
    scratch_pool_id=<optimized out>, bytes=4400, force_shrink=force_shrink@entry=false) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:441
#10 0x00000000004ccf36 in Kokkos::Impl::ParallelFor<__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int>, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>::ParallelFor(__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&) (this=0x7fffffff3ac0, arg_functor=..., arg_policy=...)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:567
#11 0x0000000000481fce in Kokkos::Impl::construct_with_shared_allocation_tracking_disabled<Kokkos::Impl::ParallelFor<__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int>, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&>(__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&) ()
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:665
#12 Kokkos::parallel_for<Kokkos::TeamPolicy<Kokkos::Cuda>, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int>, void>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::TeamPolicy<Kokkos::Cuda> const&, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&) (functor=..., policy=..., str=...)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Kokkos_Parallel.hpp:142
#13 Kokkos::parallel_for<Kokkos::TeamPolicy<Kokkos::Cuda>, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> >(Kokkos::TeamPolicy<Kokkos::Cuda> const&, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(bool), &(int Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)), 2u>, void (Kokkos::Impl::CudaTeamMember), Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>*, Kokkos::DynRankView<double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >, int> const&, std::enable_if<Kokkos::is_execution_policy<Kokkos::TeamPolicy<Kokkos::Cuda> >::value, void>::type*) (functor=..., policy=...)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Kokkos_Parallel.hpp:155
#14 Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> > (verbose=<optimized out>)
    at /ascldap/users/ndellin/Trilinos/packages/intrepid2/unit-test/Discretization/Basis/HCURL_HEX_In_FEM/test_02.hpp:125
#15 0x0000000000425576 in main (argc=<optimized out>, argv=<optimized out>)
    at /ascldap/users/ndellin/Trilinos/Build/SemsCuda1180Gcc1140OpenMPI416/packages/intrepid2/unit-test/Discretization/Basis/HCURL_HEX_In_FEM/test_02_CUDA_DOUBLE_DOUBLE.cpp:47

cuda-gdb

CUDA Exception: Warp Illegal Instruction
The exception was triggered at PC 0x0 (Kokkos_Cuda_KernelLaunch.hpp:85)

Thread 1 "Intrepid2_unit-" received signal CUDA_EXCEPTION_4, Warp Illegal Instruction.
[Switching focus to CUDA kernel 1, grid 15, block (0,0,0), thread (0,0,0), device 0, sm 0, warp 1, lane 0]
Kokkos::Impl::SharedAllocationRecord<void, void>::tracking_enabled () at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:130
130     KOKKOS_IF_ON_DEVICE(return 0;)
(cuda-gdb) bt
#0  Kokkos::Impl::SharedAllocationRecord<void, void>::tracking_enabled () at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:130
#1  0x00007fff73c42ee0 in Kokkos::DynRankView<double, Kokkos::ScratchMemorySpace<Kokkos::Cuda>, Kokkos::MemoryTraits<1u> >::DynRankView (this=0x7fff8ffff378, arg_space=..., 
    arg_N0=70, arg_N1=18446744073709551615, arg_N2=18446744073709551615, arg_N3=18446744073709551615, arg_N4=18446744073709551615, arg_N5=18446744073709551615, 
    arg_N6=18446744073709551615, arg_N7=18446744073709551615) at /ascldap/users/ndellin/Trilinos/packages/kokkos/containers/src/Kokkos_DynRankView.hpp:892
#2  0x00007fff878518b0 in Intrepid2::Basis_HCURL_HEX_In_FEM<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace>, double, double>::getValues (this=0x73c8025000007fff, 
    outputValues=0x3ff00007fff, inputPoints=0x7ffffffff, operatorType=4294967295, team_member=..., scratchStorage=..., subcellDim=-1, subcellOrdinal=-1)
    at /ascldap/users/ndellin/Trilinos/packages/intrepid2/src/Discretization/Basis/Intrepid2_HCURL_HEX_In_FEMDef.hpp:628
#3  0x00007fff73c90ce0 in Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)::{lambda(Kokkos::Impl::CudaTeamMember)#1}::operator()(Kokkos::Impl::CudaTeamMember) const (this=0x3ff <_INTERNAL_cc6f15be_24_Kokkos_Cuda_Instance_cpp_f3693ba3::Kokkos::Impl::cuda_abort(char const*)+1023>, team_member=...)
    at /ascldap/users/ndellin/Trilinos/packages/intrepid2/unit-test/Discretization/Basis/HCURL_HEX_In_FEM/test_02.hpp:91
#4  0x00007fff73c8e670 in Kokkos::Impl::ParallelFor<Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)::{lambda(Kokkos::Impl::CudaTeamMember)#1}, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>::exec_team<void> (this=0x9100001000007fff, member=...)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:482
#5  0x00007fff73c8bf00 in Kokkos::Impl::ParallelFor<Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)::{lambda(Kokkos::Impl::CudaTeamMember)#1}, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>::operator()() const (this=0x9100007000007fff)
    at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:505
#6  0x00007fff73c8a280 in Kokkos::Impl::cuda_parallel_launch_local_memory<Kokkos::Impl::ParallelFor<Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)::{lambda(Kokkos::Impl::CudaTeamMember)#1}, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda> >(Kokkos::Impl::ParallelFor<Intrepid2::Test::HCURL_HEX_In_FEM_Test02<double, double, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >(bool)::{lambda(Kokkos::Impl::CudaTeamMember)#1}, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>)
   <<<(5,1,1),(1,10,1)>>> (driver=...) at /ascldap/users/ndellin/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_KernelLaunch.hpp:87

Edit: updated the stacktrace after rebuilding with debug symbols

trilinos-autotester commented 2 weeks ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 2 weeks ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 713
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc

Build Num: 763
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 764
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_clang

Build Num: 762
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4721
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda

Build Num: 761
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_intel

Build Num: 682
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda-uvm

Build Num: 761
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 2 weeks ago

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

trilinos-autotester commented 2 weeks ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 715
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc

Build Num: 765
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 766
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_clang

Build Num: 764
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4723
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda

Build Num: 763
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_intel

Build Num: 684
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda-uvm

Build Num: 763
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 2 weeks ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 715
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc

Build Num: 765
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 766
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_clang

Build Num: 764
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4723
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda

Build Num: 763
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_intel

Build Num: 684
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

Build Information

Test Name: PR_cuda-uvm

Build Num: 763
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	c912581f73a7c7c30f0ca7618e589056a3d672cb
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	55315cdd59f28607f450b2a8ddc7b62e18315f38

CDash Test Results for PR# 13571.

Wiki: How to Reproduce PR Testing Builds and Errors.

trilinos-autotester commented 2 weeks ago

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

trilinos-autotester commented 2 weeks ago

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 724
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_gcc

Build Num: 774
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 775
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_clang

Build Num: 773
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4732
Status: STARTED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_cuda

Build Num: 772
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_intel

Build Num: 693
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_cuda-uvm

Build Num: 772
Status: STARTED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ndellingwood

trilinos-autotester commented 2 weeks ago

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: PR_gcc-openmpi-openmp

Build Num: 724
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_gcc

Build Num: 774
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_gcc-openmpi_debug

Build Num: 775
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_clang

Build Num: 773
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: Trilinos_PR_python3

Build Num: 4732
Status: PASSED

Jenkins Parameters

Parameter Name	Value
BLOCKING_BUILD	false
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_cuda

Build Num: 772
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8-gpu
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_intel

Build Num: 693
Status: FAILED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-intel-2021.3-sems-openmpi-4.1.6_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

Build Information

Test Name: PR_cuda-uvm

Build Num: 772
Status: PASSED

Jenkins Parameters

Parameter Name	Value
FORCE_CLEAN	true
GENCONFIG_BUILD_NAME	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS
PULLREQUESTNUM	13571
PULLREQUEST_CDASH_TRACK	Pull Request
TEST_REPO_ALIAS	TRILINOS
TRILINOS_NODE_LABEL	rhel8
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	1b51cb835debf2cc207456dfd1df8fa06810659c
TRILINOS_SRN_CONFIG	true
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	4e6e2ca89df63eb0050d822a60b575e2c0ddcd9c

CDash Test Results for PR# 13571.

Wiki: How to Reproduce PR Testing Builds and Errors.

ndellingwood commented 2 weeks ago

@sebrowne adding note for occurrence of nvcc error : 'ptxas' died due to signal 11 (Invalid memory reference) in the Cuda job, similar to what occurs with the nightly kokkos integration build https://trilinos-cdash.sandia.gov/viewBuildError.php?buildid=1950837

trilinos / Trilinos

Kokkos prerelease test #13571

Motivation

Related Issues

Stakeholder Feedback

Testing

Build Information

Test Name: PR_gcc-openmpi-openmp

Jenkins Parameters

Build Information

Test Name: PR_gcc

Jenkins Parameters

Build Information

Test Name: PR_gcc-openmpi_debug

Jenkins Parameters

Build Information

Test Name: PR_clang

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_python3

Jenkins Parameters

Build Information

Test Name: PR_cuda

Jenkins Parameters

Build Information

Test Name: PR_intel

Jenkins Parameters

Build Information

Test Name: PR_cuda-uvm

Jenkins Parameters

Using Repos:

Build Information

Test Name: PR_gcc-openmpi-openmp

Jenkins Parameters

Build Information

Test Name: PR_gcc

Jenkins Parameters

Build Information

Test Name: PR_gcc-openmpi_debug

Jenkins Parameters

Build Information

Test Name: PR_clang

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_python3

Jenkins Parameters

Build Information

Test Name: PR_cuda

Jenkins Parameters

Build Information

Test Name: PR_intel

Jenkins Parameters

Build Information

Test Name: PR_cuda-uvm

Jenkins Parameters

Build Information

Test Name: PR_gcc-openmpi-openmp

Jenkins Parameters

Build Information

Test Name: PR_gcc

Jenkins Parameters

Build Information

Test Name: PR_gcc-openmpi_debug

Jenkins Parameters

Build Information

Test Name: PR_clang

Jenkins Parameters

Build Information

Test Name: Trilinos_PR_python3

Jenkins Parameters

Build Information

Test Name: PR_cuda

Jenkins Parameters

Build Information

Test Name: PR_intel

Jenkins Parameters

Build Information

Test Name: PR_cuda-uvm

Jenkins Parameters

Using Repos: