search

adamantine-sim / adamantine

Software to simulate heat transfer for additive manufacturing
https://adamantine-sim.github.io/adamantine/
Other
39 stars 10 forks source link

Adamantine hangs during ArborX call for the IMTS mold mesh #316

Closed stvdwtt closed 1 month ago

stvdwtt commented 1 month ago

For the particular input files below, rank 0 of adamantine hangs in DataAssimilator::update_covariance_sparsity_pattern.

It happens during this ArborX call: https://github.com/adamantine-sim/adamantine/blob/36131aaaa13433d2d5b6cb4b998e6c5d8f8862f8/source/DataAssimilator.cc#L541

The backtrace is rather long:

#1  0x000055b51eddfabe in ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >::operator()<ArborX::Details::PairIndexVolume<ArborX::Box> > (
    this=0x7ffd1d31f118, predicate=..., value=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsCrsGraphWrapperImpl.hpp:92
#2  0x000055b51ee3077a in ArborX::Details::invoke_callback_and_check_early_exit<ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> > const&, ArborX::PredicateWithAttachment<ArborX::Intersects<ArborX::Sphere>, int> const&, ArborX::Details::PairIndexVolume<ArborX::Box> const&> (callback=..., predicate=..., primitive=...)
    at /opt/arborx/include/ArborX/details/ArborX_Callbacks.hpp:166
#3  0x000055b51ee2e6a7 in ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, --Type <RET> for more, q to quit, c to continue without paging--c
ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::operator()<ArborX::PredicateWithAttachment<ArborX::Intersects<ArborX::Sphere>, int> > (this=0x7ffd1d31f090, predicate=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsTreeTraversal.hpp:110
#4  0x000055b51ee2b0a4 in ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::operator() (this=0x7ffd1d31f090, queryIndex=6905) at /opt/arborx/include/ArborX/details/ArborX_DetailsTreeTraversal.hpp:98
#5  0x000055b51ee24933 in Kokkos::Impl::ParallelFor<ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>, Kokkos::RangePolicy<Kokkos::Serial, ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::FullTree>, Kokkos::Serial>::exec<ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::FullTree> (this=0x7ffd1d31f090) at /opt/trilinos/include/Serial/Kokkos_Serial_Parallel_Range.hpp:46
#6  0x000055b51ee182dc in Kokkos::Impl::ParallelFor<ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>, Kokkos::RangePolicy<Kokkos::Serial, ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::FullTree>, Kokkos::Serial>::execute (this=0x7ffd1d31f090) at /opt/trilinos/include/Serial/Kokkos_Serial_Parallel_Range.hpp:52
#7  0x000055b51ee07e06 in Kokkos::parallel_for<Kokkos::RangePolicy<Kokkos::Serial, ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::FullTree>, ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>, void> (str="ArborX::TreeTraversal::spatial", policy=..., functor=...) at /opt/trilinos/include/Kokkos_Parallel.hpp:144
#8  0x000055b51edf5076 in ArborX::Details::TreeTraversal<ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> >, ArborX::Details::SpatialPredicateTag>::TreeTraversal<Kokkos::Serial> (this=0x7ffd1d31f2c0, space=..., bvh=..., predicates=..., callback=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsTreeTraversal.hpp:65
#9  0x000055b51eddfc71 in ArborX::Details::traverse<Kokkos::Serial, ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> > > (space=..., bvh=..., predicates=..., callback=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsTreeTraversal.hpp:554
#10 0x000055b51edca6b9 in ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>::query<Kokkos::Serial, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::InsertGenerator<ArborX::Details::FirstPassNoBufferOptimizationTag, ArborX::Details::PermutedData<Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, true>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::PermutedData<Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, false> > > (this=0x7ffd1d31ff08, space=..., predicates=..., callback=..., policy=...) at /opt/arborx/include/ArborX/ArborX_LinearBVH.hpp:390
#11 0x000055b51edaefb7 in ArborX::Details::CrsGraphWrapperImpl::queryImpl<Kokkos::Serial, ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, Kokkos::View<unsigned int*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > > (space=..., tree=..., predicates=..., callback=..., out=..., offset=..., permute=..., buffer_status=ArborX::Details::PreallocationNone) at /opt/arborx/include/ArborX/details/ArborX_DetailsCrsGraphWrapperImpl.hpp:186
#12 0x000055b51ed8e068 in ArborX::Details::CrsGraphWrapperImpl::queryDispatch<ArborX::Details::SpatialPredicateTag, ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>, Kokkos::Serial, Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank> > (tree=..., space=..., predicates=..., callback=..., out=..., offset=..., policy=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsCrsGraphWrapperImpl.hpp:408
#13 0x000055b51ed6c98c in ArborX::BasicBoundingVolumeHierarchy<Kokkos::HostSpace, ArborX::Details::PairIndexVolume<ArborX::Box>, ArborX::Details::DefaultIndexableGetter, ArborX::Box>::query<Kokkos::Serial, Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, ArborX::Details::LegacyCallbackWrapper<ArborX::Details::DefaultCallbackWithRank>, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>&, Kokkos::View<int*, Kokkos::HostSpace>&> (this=0x7ffd1d31ff08, space=..., predicates=..., callback_or_view=..., view=...) at /opt/arborx/include/ArborX/ArborX_LinearBVH.hpp:105
#14 0x000055b51ed4d443 in ArborX::BoundingVolumeHierarchy<Kokkos::HostSpace>::query<Kokkos::Serial, Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, ArborX::Details::DefaultCallbackWithRank const&, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>&, Kokkos::View<int*, Kokkos::HostSpace>&> (this=0x7ffd1d31ff08, space=..., predicates=..., callback=..., out=..., offset=...) at /opt/arborx/include/ArborX/ArborX_LinearBVH.hpp:200
#15 0x000055b51ed35e34 in ArborX::query<ArborX::BoundingVolumeHierarchy<Kokkos::HostSpace>, Kokkos::Serial, Kokkos::View<ArborX::Intersects<ArborX::Sphere>*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >, ArborX::Details::DefaultCallbackWithRank const&, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>&, Kokkos::View<int*, Kokkos::HostSpace>&> (tree=..., space=..., predicates=..., callback_or_view=..., view=...) at /opt/arborx/include/ArborX/ArborX_CrsGraphWrapper.hpp:29
#16 0x000055b51ed1ffa8 in ArborX::Details::DistributedTreeImpl<Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::queryDispatch<ArborX::DistributedTree<Kokkos::HostSpace>, Kokkos::Serial, dealii::ArborXWrappers::SphereIntersectPredicate, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace>, ArborX::Details::DefaultCallbackWithRank> (tree=..., space=..., queries=..., callback=..., out=..., offset=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsDistributedTreeImpl.hpp:625
#17 0x000055b51ed0c6c0 in ArborX::Details::DistributedTreeImpl<Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::queryDispatch<ArborX::DistributedTree<Kokkos::HostSpace>, Kokkos::Serial, dealii::ArborXWrappers::SphereIntersectPredicate, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::HostSpace> > (tree=..., space=..., queries=..., values=..., offset=...) at /opt/arborx/include/ArborX/details/ArborX_DetailsDistributedTreeImpl.hpp:138
#18 0x000055b51ecfccc6 in ArborX::DistributedTree<Kokkos::HostSpace>::query<Kokkos::Serial, dealii::ArborXWrappers::SphereIntersectPredicate, Kokkos::View<Kokkos::pair<int, int>*, Kokkos::HostSpace>&, Kokkos::View<int*, Kokkos::HostSpace>&> (this=0x7ffd1d31fea0, space=..., predicates=...) at /opt/arborx/include/ArborX/ArborX_DistributedTree.hpp:97
#19 0x000055b51ecf0778 in dealii::ArborXWrappers::DistributedTree::query<dealii::ArborXWrappers::SphereIntersectPredicate> (this=0x7ffd1d31fea0, queries=...) at /opt/dealii/include/deal.II/arborx/distributed_tree.h:110
#20 0x000055b51ecdf139 in adamantine::DataAssimilator::update_covariance_sparsity_pattern<3> (this=0x7ffd1d320f60, dof_handler=..., parameter_size=0) at /home/adamantine/source/DataAssimilator.cc:541
#21 0x000055b51e952668 in run_ensemble<3, 0, adamantine::SolidLiquid, dealii::MemorySpace::Host> (global_communicator=@0x7ffd1d322578: 0x55b52348a800 <ompi_mpi_comm_world>, database=..., timers=std::vector of length 14, capacity 16 = {...}) at /home/adamantine/application/adamantine.hh:2040
#22 0x000055b51e88c609 in main (argc=3, argv=0x7ffd1d32f068) at /home/adamantine/application/adamantine.cc:709

Inputs: failing_example.zip

For the three ensemble members in his failing test case, I've seen it hang for 1, 3, and 9 MPI ranks. Fortunately, that means it can be debugged with serial tests.

I wonder if this and #314 have the same root cause.

@Rombur, can you take a look?

Rombur commented 1 month ago

@stvdwtt the build_time_log.txt and the .csv files are missing

stvdwtt commented 1 month ago

Sorry about that, here they are: observations.zip

Rombur commented 1 month ago

For me the code crashes inside ArborX after using all the memory available. I think that the localization cutoff distance is too large. There are 58140 supports points but the for first 1000 points, ArborX finds 22091450 points inside the cutoff radius. If the cutoff distance is physical, we need to rethink that part of the code.

stvdwtt commented 1 month ago

Closing since this isn't an ArborX issue, it's a user-issue.