Open ndellingwood opened 2 years ago
The batched gemm tests do take more cycles than our other unit-tests; I suggest increasing the timeout.
@ndellingwood: Are the batched gemm timeouts a recent regression or are you running these tests for the first time with clang >= 10 + cuda?
@ndellingwood: Are the batched gemm timeouts a recent regression or are you running these tests for the first time with clang >= 10 + cuda?
@e10harvey these tests had passed, though I can't recall a previous date / sha to give better info
I'm going to rerun the tests toggling Kokkos_ENABLE_COMPLEX_ALIGN
, this had an impact on the https://github.com/kokkos/kokkos/issues/5312 and so this may be an underlying Kokkos issue. Will post back once I finish testing
I tested builds with -DKokkos_ENABLE_COMPLEX_ALIGN=ON
and -DKokkos_ENABLE_COMPLEX_ALIGN=OFF
, the cuda.sparse_spgemm_double_int_size_t_TestExecSpace
and cuda.sparse_block_spgemm_double_int_size_t_TestExecSpace
tests fail in either case.
I rebuilt with -DKokkos_ENABLE_DEBUG=ON
and -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=ON
, but there was no additional useful diagnostic info beyond the output in the OP.
Also seeing these warning with this build:
ptxas warning : Unresolved extern variable '_ZN6Kokkos12_GLOBAL__N_13ALLE' in whole program compilation, ignoring extern qualifier
Demangled _ZN6Kokkos12_GLOBAL__N_13ALLE
[ndellin@kokkos-dev-2 Clang10Cuda101Sems-aligntest]$ c++filt -t _ZN6Kokkos12_GLOBAL__N_13ALLE
Kokkos::(anonymous namespace)::ALL
Yeah, that happened a bunch with OpenMP Target too, I will ask about it on the Kokkos channel, I'm not sure it's related though... Also I have a build going on Kokkos-dev2 so should be able to assess this problem soon.
@lucbv reproduced the failure and this is present since at least the 3.6.00 release; removing blocker on 3.7.00 promotion
In builds clang+cuda builds (e.g. clang/10+cuda/10.1, clang/13+cuda/11.7 tested) the following unit tests are failing on the develop and release-candidate-3.7.00 branches
sparse_cuda:
batched_dla_cuda: timeout
Reproducer (kokkos-dev-2):