Closed ndellingwood closed 1 month ago
The bug is in PHX::ViewOfViews2
.
Just like you have setView()
, you'd need to unset them, possibly as part of the destructor.
Otherwise the VoV destructor will attempt to delete individual inner views within a kernel and won't be able to acquire the lock since the parallel region is already active.
It is covered in https://kokkos.org/kokkos-core-wiki/ProgrammingGuide/View.html#i-really-want-a-view-of-views-what-do-i-do, you need to destroy the inner views yourself "on the host, outside of a parallel region".
The bug is in
PHX::ViewOfViews2
. Just like you havesetView()
, you'd need to unset them, possibly as part of the destructor. Otherwise the VoV destructor will attempt to delete individual inner views within a kernel and won't be able to acquire the lock since the parallel region is already active.
ViewOfViews2 was removed from the code base a couple of weeks ago when fixing #13203 . Is this failure using an old version of trilinos by any chance?
As far as I know, Nathan is testing against Trilinos develop every night. I see the issue in the current develop https://github.com/trilinos/Trilinos/blob/78784348ce1b5888d0117a25e79f8f0fb43c3069/packages/phalanx/src/design/Phalanx_KokkosViewOfViews.hpp#L205-L211
Admittedly I did not test, just reading the code and looking at the bug report.
The nightly testing with this failure occurring was most recently tested with sha a29a8fd366cf1b842f6c4bd1b9c395b417804fb7, which contained the fixes from #13203
However, the stacktrace I posted in the OP was from a manual rerun where I was bisecting the Trilinos shas and rolled back to an older sha (prior to PR #13203 ), sorry for the confusion there. I'll rebuild+rerun to add an updated stacktrace
As far as I can tell the current code will hang. I don't see where that would be mitigated.
The inner views are set here
https://github.com/trilinos/Trilinos/blob/78784348ce1b5888d0117a25e79f8f0fb43c3069/packages/phalanx/src/design/tPhalanxViewOfViews.cpp#L105-L108
and they are not being unset/destructed before the ~ViewOfViews2()
is called.
That explains it. FYI - the file with the ViewOfViews2 object is not used anymore. It's in a folder that is not a part of the source code that is compiled or installed. I saved off an old version of the file in #13203 to the design directory to remember to explore something else at a later time. The current v-of-v implementation is in the src directory (not the src/design directory): https://github.com/trilinos/Trilinos/blob/78784348ce1b5888d0117a25e79f8f0fb43c3069/packages/phalanx/src/Phalanx_KokkosViewOfViews.hpp
Here's the correct stacktrace after aborting the test:
...
5. PhalanxViewOfViews_WrapperExample_UnitTest ... [Passed] (0.00154 sec)
6. PhalanxViewOfViews_CreateHostHost_UnitTest ... [Passed] (0.000519 sec)
Thread 1 "Phalanx_tViewOf" received signal SIGINT, Interrupt.
0x0000200001ab79f8 in __lll_lock_wait () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-236.el8.ppc64le libatomic-8.5.0-10.el8.ppc64le libgcc-8.5.0-10.el8.ppc64le libgfortran-8.5.0-10.el8.ppc64le libgomp-8.5.0-10.el8.ppc64le libquadmath-8.5.0-10.el8.ppc64le libstdc++-8.5.0-10.el8.ppc64le nvidia-driver-cuda-libs-530.30.02-1.el8.ppc64le
(gdb) bt
#0 0x0000200001ab79f8 in __lll_lock_wait () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
#1 0x0000200001aaceec in pthread_mutex_lock () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
#2 0x0000000010187984 in void Kokkos::Tools::Experimental::Impl::profile_fence_event<Kokkos::OpenMP, Kokkos::OpenMP::impl_static_fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::Tools::Experimental::SpecialSynchronizationCases, Kokkos::OpenMP::impl_static_fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1} const&) [clone .isra.79] [clone .constprop.109] ()
#3 0x000000001015d22c in Kokkos::fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#4 0x000000001017a18c in Kokkos::HostSpace::deallocate(char const*, void*, unsigned long, unsigned long) const ()
#5 0x000000001017a83c in Kokkos::Impl::SharedAllocationRecordCommon<Kokkos::HostSpace>::~SharedAllocationRecordCommon() ()
#6 0x000000001002e9c8 in Kokkos::Impl::SharedAllocationRecord<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::OpenMP, double, true> >::~SharedAllocationRecord() ()
#7 0x0000000010025464 in void Kokkos::Impl::deallocate<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::OpenMP, double, true> >(Kokkos::Impl::SharedAllocationRecord<void, void>*) ()
#8 0x0000000010185044 in Kokkos::Impl::SharedAllocationRecord<void, void>::decrement(Kokkos::Impl::SharedAllocationRecord<void, void>*) ()
#9 0x0000000010025fd4 in std::enable_if<!std::is_same<Kokkos::RangePolicy<Kokkos::OpenMP, Kokkos::IndexType<long>, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>::DestroyTag>::schedule_type::type, Kokkos::Dynamic>::value, void>::type Kokkos::Impl::ParallelFor<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>, Kokkos::RangePolicy<Kokkos::OpenMP, Kokkos::IndexType<long>, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>::DestroyTag>, Kokkos::OpenMP>::execute_parallel<Kokkos::RangePolicy<Kokkos::OpenMP, Kokkos::IndexType<long>, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>::DestroyTag> >() const [clone ._omp_fn.46] ()
#10 0x0000200001a04ca8 in GOMP_parallel () from /lib64/libgomp.so.1
#11 0x0000000010058d14 in void Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>::parallel_for_implementation<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false>::DestroyTag>() ()
#12 0x0000000010059104 in void Kokkos::Impl::deallocate<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>, false> >(Kokkos::Impl::SharedAllocationRecord<void, void>*)
()
#13 0x0000000010185044 in Kokkos::Impl::SharedAllocationRecord<void, void>::decrement(Kokkos::Impl::SharedAllocationRecord<void, void>*) ()
#14 0x0000000010030970 in Kokkos::Impl::ViewTracker<Kokkos::View<Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>**, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::Experimental::EmptyViewHooks> >::operator=(Kokkos::Impl::ViewTracker<Kokkos::View<Kokkos::View<Sacado::Fad::Exp::GeneralFad<Sacado::Fad::Exp::DynamicStorage<double, double> >**, Kokkos::LayoutRight, Kokkos::OpenMP>**, Kokkos::LayoutRight, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::Experimental::EmptyViewHooks> > const&) ()
#15 0x00000000100234b0 in PhalanxViewOfViews_FadAndAssignment_UnitTest::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const ()
#16 0x00000000101501a8 in Teuchos::UnitTestBase::runUnitTest(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&) const ()
#17 0x0000000010153a94 in Teuchos::UnitTestRepository::runUnitTestImpl(Teuchos::UnitTestBase const&, Teuchos::basic_FancyOStream<char, std::char_traits<char> >&) ()
#18 0x0000000010156fc8 in Teuchos::UnitTestRepository::runUnitTests(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&) ()
#19 0x00000000101580e0 in Teuchos::UnitTestRepository::runUnitTestsFromMain(int, char**) ()
#20 0x000000001000d16c in main ()
I was able to replicate in a gcc 12 build. It's hanging in the default move constructor for the VoV object. Should have a fix up soon.
Proposed resolution that fixes the deadlock but lacks the safeguard if users hold on to a copy of the underlying vovs. phalanx_vov.patch
(edit: had messed up the bounds in the inner view deleter)
@rppawlo @dalg24 I confirmed the patch Damien provided above resolved the timeout issue for me
[ndellin@weaver8 ViewOfViews]$ ctest -R Phalanx_tViewOfViews_MPI_1 -V
...
1: Running unit tests ...
1:
1: 0. PhalanxViewOfViews_ViewOfView3_DefaultStreamInitialize_UnitTest ... [Passed] (0.0157 sec)
1: 1. PhalanxViewOfViews_ViewOfView3_DefaultStreamCtor_UnitTest ... [Passed] (0.0143 sec)
1: 2. PhalanxViewOfViews_ViewOfView3_UserStreamCtor_UnitTest ... Using partition_space, concurrency=8
1: [Passed] (0.0127 sec)
1: 3. PhalanxViewOfViews_ViewOfView3_UserStreamInitialize_UnitTest ... Using partition_space, concurrency=8
1: [Passed] (0.0125 sec)
1: 4. PhalanxViewOfViews_ViewOfView3_DefaultCtorDtor_UnitTest ... [Passed] (2.71e-06 sec)
1: 5. PhalanxViewOfViews_WrapperExample_UnitTest ... [Passed] (0.000682 sec)
1: 6. PhalanxViewOfViews_CreateHostHost_UnitTest ... [Passed] (0.000744 sec)
1: 7. PhalanxViewOfViews_FadAndAssignment_UnitTest ... [Passed] (0.00139 sec)
1: 8. PhalanxViewOfViews_FadHierarchicMDRangeBug_UnitTest ...
1: [Passed] (0.00012 sec)
1:
1: Total Time: 0.0585 sec
1:
1: Summary: total = 9, run = 9, passed = 9, failed = 0
1:
1: End Result: TEST PASSED
1/1 Test #1: Phalanx_tViewOfViews_MPI_1 ....... Passed 0.76 sec
The following tests passed:
Phalanx_tViewOfViews_MPI_1
100% tests passed, 0 tests failed out of 1
Label Time Summary:
Phalanx = 0.76 sec*proc (1 test)
Total Test time (real) = 0.80 sec
Thanks for the patch @dalg24 ! That's some neat code! I really want to preserve the safeguards. They have been important in the past with the app teams. I've got an alternate solution that seems to be working - it's up at https://github.com/trilinos/Trilinos/pull/13287 if you want to take a look. I will use some of your ideas in that patch in the future.
I think you could preserve the runtime check by doing something along these lines. phalanx_vov_v2.patch
FWIW it passes the tests from your PR. In my opinion, implementing all special operators like you did is higher risk to mess things up and more costly to maintain.
Agreed. I'm pulling in your changes and testing against some internal applications. Will do some further cleanup and push later today.
@ndellingwood - changes have been pushed into Trilinos develop if you want to test.
Thanks @rppawlo ! After those changes, I'm seeing timeouts (cuda/11.2.2 build with UVM) in the Phalanx_tViewOfViews_MPI_1
unit test, I can post the configure for weaver if helpful. Sorry for the whack-a-mole :(
The gcc/8.5.0 OpenMP build is back to passing
@ndellingwood - I've reproduced it. Working on a fix. These are the flags I used to enable UVM. Is this what you use?
-D Kokkos_ENABLE_CUDA_UVM=ON \
-D KokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON \
-D Tpetra_ENABLE_CUDA_UVM=ON \
@rppawlo yes to the Kokkos and KokkosKernels; for Tpetra I use -D Tpetra_ALLOCATE_IN_SHARED_SPACE=ON
Linking #13300 for the Cuda+UVM PR
@rppawlo I'm still seeing the Phalanx_tViewOfViews_MPI_1
timeout for Cuda+UVM integration builds after merge of #13300, here are some reproducer details from the job on Weaver:
Test output:
...
Running unit tests ...
0. PhalanxViewOfViews_ViewOfView_DefaultStreamInitialize_UnitTest ... [Passed] (0.0122 sec)
1. PhalanxViewOfViews_ViewOfView_DefaultStreamCtor_UnitTest ... [Passed] (0.0117 sec)
2. PhalanxViewOfViews_ViewOfView_UserStreamCtor_UnitTest ... Using partition_space, concurrency=163840
[Passed] (0.0125 sec)
3. PhalanxViewOfViews_ViewOfView_UserStreamInitialize_UnitTest ... Using partition_space, concurrency=163840
[Passed] (0.0124 sec)
4. PhalanxViewOfViews_ViewOfView_DefaultCtorDtor_UnitTest ... [Passed] (5.42e-06 sec)
5. PhalanxViewOfViews_WrapperExample_UnitTest ... [Passed] (0.0011 sec)
0% tests passed, 1 tests failed out of 1
Subproject Time Summary:
Phalanx = 1500.12 sec*proc (1 test)
Total Test time (real) = 1501.07 sec
The following tests FAILED:
813 - Phalanx_tViewOfViews_MPI_1 (Timeout)
Weaver reproducer (rhel8 queue): SHA: 665bf893489919d69b3a31fd5e3313e831dda117
# Repo prep
git clone -b develop https://github.com/trilinos/Trilinos.git
TRILINOS_DIR=$PWD/Trilinos
git clone -b develop https://github.com/kokkos/kokkos.git
KOKKOS_DIR=$PWD/kokkos
git clone -b develop https://github.com/kokkos/kokkos-kernels.git
KOKKOSKERNELS_DIR=$PWD/kokkos-kernels
cd $TRILINOS_DIR
ln -s ${KOKKOS_DIR}/kokkos kokkos
ln -s ${KOKKOSKERNELS_DIR}/kokkos-kernels kokkos-kernels
cd -
mkdir -p Build
cd Build
# Environment and configuration
export TRILINOS_DIR=<path-to-trilinos>
# Interactive compute node
bsub -Is -n 1 -q rhel8 -gpu "num=4" bash
source /etc/profile.d/modules.sh
source /projects/ppc64le-pwr9-rhel8/legacy-env.sh
module purge
module load git cmake/3.23.1 cuda/11.2.2/gcc/8.3.1 openmpi/4.1.1/gcc/8.3.1/cuda/11.2.2 openblas/0.3.18/gcc/8.3.1
module list
export OMPI_CXX=$KOKKOS_DIR/bin/nvcc_wrapper
cmake \
-D CMAKE_CXX_COMPILER="`which mpicxx`" \
-D CMAKE_C_COMPILER="`which mpicc`" \
-D CMAKE_Fortran_COMPILER="`which mpifort`" \
-D CMAKE_BUILD_TYPE:STRING=RELEASE \
-D CMAKE_CXX_STANDARD=17 \
-D BUILD_SHARED_LIBS=ON \
-D CMAKE_INSTALL_PREFIX=$PWD/install \
-D TPL_ENABLE_CUDA:STRING=ON \
-D TPL_ENABLE_MPI:STRING=ON \
-D MPI_BASE_DIR:PATH="$OPENMPI_ROOT" \
-D MPI_BIN_DIR:PATH="$OPENMPI_BIN" \
-D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-map-by;socket:PE=4" \
-D TPL_ENABLE_BLAS:STRING=ON \
-D BLAS_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
-D BLAS_LIBRARY_NAMES:STRING="openblas" \
-D TPL_ENABLE_LAPACK:STRING=ON \
-D LAPACK_INCLUDE_DIRS:FILEPATH="$OPENBLAS_ROOT/include" \
-D LAPACK_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
-D LAPACK_LIBRARY_NAMES:STRING="openblas" \
-D Trilinos_ENABLE_COMPLEX=ON \
-D Trilinos_ENABLE_TESTS=OFF \
-D Trilinos_ENABLE_EXAMPLES=OFF \
-D Trilinos_ENABLE_Kokkos=ON \
-D Kokkos_ENABLE_CUDA=ON \
-D Kokkos_ENABLE_CUDA_LAMBDA=ON \
-D Kokkos_ENABLE_CUDA_UVM=ON \
-D Kokkos_ARCH_VOLTA70=ON \
-D Kokkos_ARCH_POWER9=ON \
-D Kokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF \
-D Kokkos_ENABLE_TESTS=ON \
-D Kokkos_ENABLE_IMPL_MDSPAN=ON \
-D Trilinos_ENABLE_KokkosKernels=ON \
-D KokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON \
-D KokkosKernels_ENABLE_TPL_CUSOLVER=ON \
-D KokkosKernels_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Tpetra=ON \
-D Tpetra_ALLOCATE_IN_SHARED_SPACE=ON \
-D Tpetra_INST_CUDA=ON \
-D Tpetra_INST_SERIAL=ON \
-D Tpetra_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Sacado=ON \
-D Sacado_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Phalanx=ON \
-D Phalanx_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Adelus=ON \
-D Adelus_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Ifpack2=ON \
-D Ifpack2_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Amesos2=OFF \
-D Trilinos_ENABLE_Stokhos=OFF \
-D Trilinos_ENABLE_MueLu=OFF \
-D Intrepid2_ENABLE_TESTS=OFF \
-D Trilinos_ENABLE_ShyLU_NodeTacho=OFF \
-D Kokkos_CoreUnitTest_Default_MPI_1_EXTRA_ARGS="--gtest_filter=-*defaultdevicetype.shared_space" \
-DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
-DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
-DTrilinos_ENABLE_INSTALLATION_TESTING=OFF \
$TRILINOS_DIR
(Most of the packages above can be disabled, just posting full config)
Edit: Adding the backtrace from gdb
[New Thread 0x200088df0890 (LWP 136065)]
0. PhalanxViewOfViews_ViewOfView_DefaultStreamInitialize_UnitTest ... [Passed] (0.0125 sec)
1. PhalanxViewOfViews_ViewOfView_DefaultStreamCtor_UnitTest ... [Passed] (0.0117 sec)
2. PhalanxViewOfViews_ViewOfView_UserStreamCtor_UnitTest ... Using partition_space, concurrency=163840
[Passed] (0.0124 sec)
3. PhalanxViewOfViews_ViewOfView_UserStreamInitialize_UnitTest ... Using partition_space, concurrency=163840
[Passed] (0.0123 sec)
4. PhalanxViewOfViews_ViewOfView_DefaultCtorDtor_UnitTest ... [Passed] (4.66e-06 sec)
5. PhalanxViewOfViews_WrapperExample_UnitTest ... [Passed] (0.0012 sec)
Thread 1 "Phalanx_tViewOf" received signal SIGINT, Interrupt.
0x0000200042d379f8 in __lll_lock_wait () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
(gdb) bt
#0 0x0000200042d379f8 in __lll_lock_wait () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
#1 0x0000200042d2ceec in pthread_mutex_lock () from /lib64/glibc-hwcaps/power9/libpthread-2.28.so
#2 0x0000200042810070 in __gthread_mutex_lock (__mutex=0x10781b00) at /usr/include/c++/8/ppc64le-redhat-linux/bits/gthr-default.h:748
#3 std::mutex::lock (this=0x10781b00) at /usr/include/c++/8/bits/std_mutex.h:103
#4 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /usr/include/c++/8/bits/std_mutex.h:162
#5 Kokkos::Serial::impl_static_fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}::operator()() const (
__closure=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Serial/Kokkos_Serial.hpp:155
#6 Kokkos::Tools::Experimental::Impl::profile_fence_event<Kokkos::Serial, Kokkos::Serial::impl_static_fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::Tools::Experimental::SpecialSynchronizationCases, Kokkos::Serial::impl_static_fence(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1} const&) (func=...,
reason=Kokkos::Tools::Experimental::GlobalDeviceSynchronization, name=...) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_Profiling.hpp:208
#7 Kokkos::Serial::impl_static_fence (name=...) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Serial/Kokkos_Serial.hpp:147
#8 Kokkos::Impl::ExecSpaceDerived<Kokkos::Serial>::static_fence (this=<optimized out>, label=...)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_ExecSpaceManager.hpp:131
#9 0x00002000427e699c in Kokkos::Impl::ExecSpaceManager::static_fence (this=0x2000428a17c0 <Kokkos::Impl::ExecSpaceManager::get_instance()::space_initializer>, name=...)
at /usr/include/c++/8/bits/unique_ptr.h:345
#10 0x00002000427e6e3c in (anonymous namespace)::fence_internal (name=...) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_Core.cpp:813
#11 0x00002000427e7108 in Kokkos::fence (name=...) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_Core.cpp:1099
#12 0x00002000427f0c20 in Kokkos::HostSpace::deallocate (this=0x11097068, arg_label=0x7fffffff8830 "a", arg_alloc_ptr=0x37b8de00, arg_alloc_size=168, arg_logical_size=40)
at /usr/include/c++/8/bits/char_traits.h:287
#13 0x00002000427f11ec in Kokkos::Impl::SharedAllocationRecordCommon<Kokkos::HostSpace>::~SharedAllocationRecordCommon (this=0x11097020, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/basic_string.h:2294
#14 0x0000000010033c88 in Kokkos::Impl::SharedAllocationRecord<Kokkos::HostSpace, void>::~SharedAllocationRecord (this=<optimized out>, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Kokkos_HostSpace.hpp:178
#15 Kokkos::Impl::SharedAllocationRecord<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, double, true> >::~SharedAllocationRecord (this=0x11097020, __in_chrg=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:400
#16 Kokkos::Impl::SharedAllocationRecord<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, double, true> >::~SharedAllocationRecord (this=0x11097020, __in_chrg=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:400
#17 0x000000001002cb84 in Kokkos::Impl::deallocate<Kokkos::HostSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, double, true> > (
record_ptr=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:375
#18 0x00002000427fa3a4 in Kokkos::Impl::SharedAllocationRecord<void, void>::decrement (arg_record=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.cpp:268
#19 0x0000000010065ff4 in Kokkos::Impl::SharedAllocationTracker::~SharedAllocationTracker (this=0x2000e0000c80, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:544
#20 Kokkos::Impl::ViewTracker<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace> >::~ViewTracker (this=0x2000e0000c80, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_ViewTracker.hpp:39
#21 Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>::~View (this=0x2000e0000c80, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Kokkos_View.hpp:1279
#22 Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::operator() (
this=0x7fffffff8b70, i=0) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/View/Kokkos_ViewAlloc.hpp:80
#23 Kokkos::Impl::ParallelFor<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>, Kokkos::RangePolicy<Kokkos::Serial, Kokkos::IndexType<long>, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::DestroyTag>, Kokkos::Serial>::exec<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::DestroyTag> (this=0x7fffffff8b70)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:46
#24 Kokkos::Impl::ParallelFor<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>, Kokkos::RangePolicy<Kokkos::Serial, Kokkos::IndexType<long>, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::DestroyTag>, Kokkos::Serial>::execute (this=0x7fffffff8b70)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Range.hpp:56
#25 Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::parallel_for_implementation<Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::DestroyTag> (this=0x37b8cbb8) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/View/Kokkos_ViewAlloc.hpp:174
#26 0x0000000010066404 in Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false>::destroy_shared_allocation (this=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/View/Kokkos_ViewAlloc.hpp:184
#27 Kokkos::Impl::deallocate<Kokkos::CudaUVMSpace, Kokkos::Impl::ViewValueFunctor<Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, false> > (record_ptr=0x37b8cb60) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:382
#28 0x00002000427fa3a4 in Kokkos::Impl::SharedAllocationRecord<void, void>::decrement (arg_record=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.cpp:268
#29 0x0000000010031300 in Kokkos::Impl::SharedAllocationTracker::~SharedAllocationTracker (this=0x37b8c420, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:544
#30 Kokkos::Impl::ViewTracker<Kokkos::View<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks> >::~ViewTracker (this=0x37b8c420, __in_chrg=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
at /home/ndellin/trilinos/Trilinos/kokkos/core/src/impl/Kokkos_ViewTracker.hpp:39
#31 Kokkos::View<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks>::~View (this=0x37b8c420, __in_chrg=<optimized out>) at /home/ndellin/trilinos/Trilinos/kokkos/core/src/Kokkos_View.hpp:1279
#32 PHX::details::ViewOfViewsDeleter::operator()<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks> (this=<optimized out>, vov=0x37b8c420) at /home/ndellin/trilinos/Trilinos/packages/phalanx/src/Phalanx_KokkosViewOfViews.hpp:77
#33 PHX::details::ViewOfViewsDeleter::operator()<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks> (this=<optimized out>, vov=0x37b8c420) at /home/ndellin/trilinos/Trilinos/packages/phalanx/src/Phalanx_KokkosViewOfViews.hpp:41
#34 std::_Sp_counted_deleter<Kokkos::View<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks>*, PHX::details::ViewOfViewsDeleter, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr_base.h:471
#35 0x000000001003c9ec in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x37b91910) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#36 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#37 0x00000000100282c8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fffffff9048, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#38 std::__shared_ptr<Kokkos::View<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fffffff9040, __in_chrg=<optimized out>)
at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#39 std::shared_ptr<Kokkos::View<Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::Experimental::EmptyViewHooks> >::~shared_ptr (this=0x7fffffff9040, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr.h:103
#40 PHX::ViewOfViews<1, Kokkos::View<double*, Kokkos::LayoutLeft, Kokkos::HostSpace>, Kokkos::CudaUVMSpace>::~ViewOfViews (this=0x7fffffff9040, __in_chrg=<optimized out>)
at /home/ndellin/trilinos/Trilinos/packages/phalanx/src/Phalanx_KokkosViewOfViews.hpp:126
#41 PhalanxViewOfViews_CreateHostHost_UnitTest::runUnitTestImpl (this=<optimized out>, out=..., success=@0x7fffffff9978: true)
at /home/ndellin/trilinos/Trilinos/packages/phalanx/test/ViewOfViews/tPhalanxViewOfViews.cpp:459
#42 0x000020002e0c8844 in Teuchos::UnitTestBase::runUnitTest (this=<optimized out>, out=...)
at /home/ndellin/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_UnitTestBase.cpp:29
#43 0x000020002e0ced64 in Teuchos::UnitTestRepository::runUnitTestImpl (unitTest=..., out=...)
at /home/ndellin/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_UnitTestRepository.cpp:506
#44 0x000020002e0d037c in Teuchos::UnitTestRepository::runUnitTests (out=...) at /home/ndellin/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_UnitTestRepository.cpp:284
#45 0x000020002e0d1650 in Teuchos::UnitTestRepository::runUnitTestsFromMain (argc=<optimized out>, argv=<optimized out>)
at /home/ndellin/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_UnitTestRepository.cpp:390
#46 0x000020002d6a797c in main (argc=<optimized out>, argv=<optimized out>) at /home/ndellin/trilinos/Trilinos/packages/phalanx/test/Utilities/Phalanx_UnitTestMain.cpp:29
#47 0x0000200043139f5c in generic_start_main.isra () from /lib64/glibc-hwcaps/power9/libc-2.28.so
#48 0x000020004313a0f4 in __libc_start_main () from /lib64/glibc-hwcaps/power9/libc-2.28.so
#49 0x0000000000000000 in ?? ()
Final fix was #13316
The nightly Cuda+UVM job I referenced above is looking good, timeout resolve :)
Test #797: Phalanx_tKokkosViewOfViews_MPI_1 ............................................................ Passed 1.20 sec
...
Test #813: Phalanx_tViewOfViews_MPI_1 .................................................................. Passed 1.46 sec
Bug Report
@trilinos/phalanx @rppawlo
Description
The Phalanx Phalanx_tViewOfViews_MPI_1 unit tests are hanging until timeout in gcc/8.5.0 builds with the OpenMP backend when tested with the Kokkos develop branch (to-be 4.4 release). This has been happening since before merge of #13203 (it was masked by other unrelated issues)
This is the stacktrace from gdb when manually aborting the job:
The hang occurs regardless of number of threads tested with (tested with 1, 4, 8 threads and hang occurred in each case) @rppawlo can you help advise?
Steps to Reproduce
Reproducer configuration (weaver testbed):
This was seen on the Weaver testbed (Power9+Volta70, though the GPU type is irrelevant). I also noticed the hangs occurring in the nightly Framework's build with gcc/8