search

STEllAR-GROUP / octotiger

Astrophysics program simulating the evolution of star systems based on the fast multipole method on adaptive Octrees
http://octotiger.stellar-group.org/
Boost Software License 1.0
48 stars 17 forks source link

Octo-Tiger segfaults or shows a floating point error while running v1309 #471

Closed diehlpk closed 5 months ago

diehlpk commented 8 months ago 

{version}: V1.9.1 (AGAS: V3.0), Git: unknown
{boost}: V1.80.0
{build-type}: release
{date}: Oct 27 2023 06:50:20
{platform}: linux
{compiler}: GNU C++ version 12.2.0
{stdlib}: GNU libstdc++ version 20220819

{stack-trace}: 10 frames:
0x45b370        : hpx::detail::spinlock<true>::lock() [0x10] in octotiger
0x4000010b76d0  : void Kokkos::Experimental::HPX::impl_bulk_plain_erased<unsigned long>(bool, bool, std::function<void (unsigned long)>&&, unsigned long, hpx::threads::thread_stacksize) const [0x50] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000008153f4  : void Kokkos::parallel_for<Kokkos::MDRangePolicy<Kokkos::Experimental::HPX, Kokkos::Rank<4u, (Kokkos::Iterate)0, (Kokkos::Iterate)0>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul> >, find_contact_discs_impl<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> >(Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, double, double, double, double, unsigned long, unsigned long, Kokkos::Array<long, 4ul, void> const&&, Kokkos::Array<long, 4ul, void> const&&)::{lambda(int, int, int, int)#1}, void>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::MDRangePolicy<Kokkos::Experimental::HPX, Kokkos::Rank<4u, (Kokkos::Iterate)0, (Kokkos::Iterate)0>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul> > const&, find_contact_discs_impl<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> >(Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, double, double, double, double, unsigned long, unsigned long, Kokkos::Array<long, 4ul, void> const&&, Kokkos::Array<long, 4ul, void> const&&)::{lambda(int, int, int, int)#1} const&) [0x2b4] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000008222e8  : void find_contact_discs_impl<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> >(Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, double, double, double, double, unsigned long, unsigned long, Kokkos::Array<long, 4ul, void> const&&, Kokkos::Array<long, 4ul, void> const&&) [0x268] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000008248c8  : timestep_t device_interface_kokkos_hydro<hpx::kokkos::executor<Kokkos::Experimental::HPX>, 0>(recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, recycler::aggregated_recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, int> const&, recycler::aggregated_recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, int> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, unsigned long, unsigned long, bool, unsigned long, double, int, int, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, double, double, double, double, Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice&, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >&, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >&) [0x148] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x400000825428  : auto launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}::operator()<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice> >(hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>&&) const [0x468] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000008259b8  : void hpx::lcos::detail::invoke_continuation_nounwrap<launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t> >(launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}&, hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>&&, hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t>&) [0x38] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000007fc194  : std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> hpx::util::detail::callable_vtable<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::threads::thread_restart_state)>::_invoke<hpx::threads::detail::thread_function_nullary<hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t>::async<true, hpx::lcos::detail::post_policy_spawner>(hpx::intrusive_ptr<hpx::lcos::detail::future_data_base<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice> >&&, hpx::lcos::detail::post_policy_spawner&&)::{lambda()#1}> >(void*, hpx::threads::thread_restart_state&&) [0x74] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x40000196d32c  : hpx::threads::coroutines::detail::coroutine_impl::operator()() [0xec] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x40000196c978  : /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so(+0xfc978) [0x40000196c978] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
{what}: Segmentation fault
8 frames:
0x40000081f0d4  : /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so(+0x5f0d4) [0x40000081f0d4] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000008209ac  : recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>::aggregated_recycled_view<unsigned long>(Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >&, unsigned long) [0x32c] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x400000824814  : timestep_t device_interface_kokkos_hydro<hpx::kokkos::executor<Kokkos::Experimental::HPX>, 0>(recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, recycler::aggregated_recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, int> const&, recycler::aggregated_recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, int> const&, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double>&, unsigned long, unsigned long, bool, unsigned long, double, int, int, recycler::aggregated_recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >, double> const&, double, double, double, double, Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice&, Allocator_Slice<double, std::allocator<double>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >&, Allocator_Slice<int, std::allocator<int>, hpx::kokkos::executor<Kokkos::Experimental::HPX> >&) [0x94] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x400000825428  : auto launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}::operator()<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice> >(hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>&&) const [0x468] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000008259b8  : void hpx::lcos::detail::invoke_continuation_nounwrap<launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t> >(launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}&, hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>&&, hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t>&) [0x38] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x4000007fc194  : std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> hpx::util::detail::callable_vtable<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::threads::thread_restart_state)>::_invoke<hpx::threads::detail::thread_function_nullary<hpx::lcos::detail::continuation<hpx::future<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice>, launch_hydro_kokkos_kernels<hpx::kokkos::executor<Kokkos::Experimental::HPX> >(hydro_computer<3, 8, physics<3> > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > const&, double, unsigned long, std::vector<hydro_state_t<std::vector<double, std::allocator<double> > >, std::allocator<hydro_state_t<std::vector<double, std::allocator<double> > > > >&)::{lambda(auto:1&&)#1}, timestep_t>::async<true, hpx::lcos::detail::post_policy_spawner>(hpx::intrusive_ptr<hpx::lcos::detail::future_data_base<Aggregated_Executor<hpx::kokkos::executor<Kokkos::Experimental::HPX> >::Executor_Slice> >&&, hpx::lcos::detail::post_policy_spawner&&)::{lambda()#1}> >(void*, hpx::threads::thread_restart_state&&) [0x74] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_hydrolib.so
0x40000196d32c  : hpx::threads::coroutines::detail::coroutine_impl::operator()() [0xec] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x40000196c978  : /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so(+0xfc978) [0x40000196c978] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
{what}: Segmentation fault
diehlpk commented 8 months ago

It seems that removing --amr_boundary_kernel_type=AMR_OPTIMIZED resolves the issue.

diehlpk commented 8 months ago

However, now we get a floating point exception.


version}: V1.9.1 (AGAS: V3.0), Git: unknown
{boost}: V1.80.0
{build-type}: release
{date}: Oct 27 2023 06:50:20
{platform}: linux
{compiler}: GNU C++ version 12.2.0
{stdlib}: GNU libstdc++ version 20220819

{stack-trace}: 17 frames:
0x4000005a03a4  : octotiger::fmm::monopole_interactions::p2m_kernel_impl_rho<sve::experimental::parallelism_v2::simd<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, sve::experimental::parallelism_v2::simd_mask<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, Kokkos::Experimental::HPX, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> >(hpx::kokkos::executor<Kokkos::Experimental::HPX>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, double, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> const&, bool)::{lambda(int, int, int)#2}::operator()(int, int, int) const [0x7c4] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_octolib.so
0x4000005a0ef4  : std::_Function_handler<void (unsigned long), Kokkos::Experimental::HPX::impl_bulk_plain<Kokkos::Impl::ParallelFor<octotiger::fmm::monopole_interactions::p2m_kernel_impl_rho<sve::experimental::parallelism_v2::simd<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, sve::experimental::parallelism_v2::simd_mask<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, Kokkos::Experimental::HPX, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> >(hpx::kokkos::executor<Kokkos::Experimental::HPX>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, double, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> const&, bool)::{lambda(int, int, int)#2}, Kokkos::MDRangePolicy<Kokkos::Experimental::HPX, Kokkos::Rank<3u, (Kokkos::Iterate)0, (Kokkos::Iterate)0>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul> >, Kokkos::Experimental::HPX>, unsigned long>(bool, bool, Kokkos::Impl::ParallelFor<octotiger::fmm::monopole_interactions::p2m_kernel_impl_rho<sve::experimental::parallelism_v2::simd<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, sve::experimental::parallelism_v2::simd_mask<double, sve::experimental::parallelism_v2::simd_abi::sve_abi>, Kokkos::Experimental::HPX, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> >(hpx::kokkos::executor<Kokkos::Experimental::HPX>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double> const&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, recycler::recycled_view<Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<double, std::allocator<double> >, double>&, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, double, octotiger::fmm::multiindex<int>, octotiger::fmm::multiindex<int>, recycler::recycled_view<Kokkos::View<int*, Kokkos::LayoutRight, Kokkos::HostSpace, Kokkos::MemoryTraits<1u> >, recycler::detail::recycle_allocator<int, std::allocator<int> >, int> const&, bool)::{lambda(int, int, int)#2}, Kokkos::MDRangePolicy<Kokkos::Experimental::HPX, Kokkos::Rank<3u, (Kokkos::Iterate)0, (Kokkos::Iterate)0>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul> >, Kokkos::Experimental::HPX> const&, unsigned long, hpx::threads::thread_stacksize) const::{lambda(unsigned long)#1}>::_M_invoke(std::_Any_data const&, unsigned long&&) [0x294] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/octotiger-0.10.0-2en7plwicmukr2hnmeddi456uvsek6q5/lib64/libhpx_octolib.so
0x4000010bc21c  : hpx::execution::experimental::detail::any_receiver_impl<hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::unique_any_sender<>, hpx::detail::bound_front<std::function<void (unsigned long)>, hpx::util::pack_c<unsigned long, 0ul>, int> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::split_receiver, void (&)()>, hpx::detail::bound_front<std::function<void (unsigned long)>, hpx::util::pack_c<unsigned long, 0ul>, int> >>::set_value() && [0x3c] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010a3c70  : void hpx::util::detail::callable_vtable<void ()>::_invoke<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::add_continuation<hpx::execution::experimental::detail::any_receiver<> >(hpx::execution::experimental::detail::any_receiver<>&&)::{lambda()#1}>(void*) [0xb0] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010a7670  : hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::set_predecessor_done() [0x70] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010be13c  : void hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::split_receiver, void (&)()>::set_value_helper<>() && [0x5c] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010be344  : hpx::execution::experimental::detail::task_function<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >::operation_state<hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::split_receiver, void (&)()> > >::finish() const [0x104] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010c8e70  : hpx::execution::experimental::detail::task_function<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >::operation_state<hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::split_receiver, void (&)()> > >::operator()() const [0x290] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x4000010a2f34  : std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> hpx::util::detail::callable_vtable<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::threads::thread_restart_state)>::_invoke<hpx::threads::detail::thread_function_nullary<hpx::execution::experimental::detail::task_function<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >::operation_state<hpx::execution::experimental::detail::then_receiver<hpx::execution::experimental::detail::split_sender<hpx::execution::experimental::detail::then_sender<hpx::execution::experimental::detail::thread_pool_bulk_sender<hpx::launch, hpx::execution::experimental::detail::schedule_from_sender<hpx::execution::experimental::unique_any_sender<>, hpx::execution::experimental::thread_pool_policy_scheduler<hpx::launch> >, hpx::util::counting_shape<unsigned long>, std::function<void (unsigned long)> >, void (&)()>, std::allocator<int>, (hpx::execution::experimental::detail::submission_type)0, hpx::execution::experimental::detail::no_scheduler>::shared_state::split_receiver, void (&)()> > > > >(void*, hpx::threads::thread_restart_state&&) [0x14] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/kokkos-develop-xqyk7mhd7uhpmxzjc7bl23ybv2ty72nh/lib64/libkokkoscore.so.4.2
0x400001a36a00  : hpx::threads::thread_data_stackless::call() [0xc0] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x400001a78ffc  : /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so(+0x208ffc) [0x400001a78ffc] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x400001a79e00  : hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_lifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::thread_func(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>) [0x3c0] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x400001a3a620  : std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_lifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >::*)(unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier>), hpx::threads::detail::scheduled_thread_pool<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_lifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo> >*, unsigned long, unsigned long, std::shared_ptr<hpx::util::barrier> > > >::_M_run() [0x40] in /vol0004/ra010008/data/u10393/spackfugaku/opt/spack/linux-rhel8-a64fx/gcc-12.2.0/hpx-1.9.1-2e66nv3zy3impgqw72uqfbn4leseekyd/lib64/libhpx_core.so
0x400001f56e3c  : /vol0004/apps/oss/spack-v0.19/opt/spack/linux-rhel8-a64fx/gcc-8.5.0/gcc-12.2.0-sxcx7kmt3qiktffgzzvrj2wmup3g32bc/lib64/libstdc++.so.6(+0xd6e3c) [0x400001f56e3c] in /vol0004/apps/oss/spack-v0.19/opt/spack/linux-rhel8-a64fx/gcc-8.5.0/gcc-12.2.0-sxcx7kmt3qiktffgzzvrj2wmup3g32bc/lib64/libstdc++.so.6
0x4000021c78b8  : /lib64/libpthread.so.0(+0x78b8) [0x4000021c78b8] in /lib64/libpthread.so.0
0x400002223afc  : /lib64/libc.so.6(+0x23afc) [0x400002223afc] in /lib64/libc.so.6
{what}: Floating point exception
diehlpk commented 8 months ago

Maybe related to #430

diehlpk commented 8 months ago

cc @srinivasyadav18

G-071 commented 8 months ago

Can you please post the full spack spec /hash-of-your-build?

The segfaults/exceptions you posted are all over the place (some in the hydro solver, some in the gravity solver). Unfortunately, those may occur if the kernel is fed with bad input data (meaning the problem might be somewhere else). Still, can you prepare a build with the spack octotiger variant ~kokkos_hpx_kernels to make sure the newer sender and receiver backend for the kernels is not the culprit as that's the only thing the crashes above have in common otherwise. Also, an additional debug run using the LEGACY kernels for a few time-steps would be useful.

diehlpk commented 8 months ago

@G-071 I will provide the spack output soon, but first some more details

{config}:
Core library:
  HPX_AGAS_LOCAL_CACHE_SIZE=4096
  HPX_HAVE_MALLOC=jemalloc
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_PREFIX (configured)=
  HPX_PREFIX=

  HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
  HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
  HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
  HPX_WITH_APEX=OFF
  HPX_WITH_ASYNC_MPI=OFF
  HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
  HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
  HPX_WITH_COROUTINE_COUNTERS=OFF
  HPX_WITH_DISTRIBUTED_RUNTIME=ON
  HPX_WITH_DYNAMIC_HPX_MAIN=ON
  HPX_WITH_GENERIC_CONTEXT_COROUTINES=ON
  HPX_WITH_IO_COUNTERS=ON
  HPX_WITH_IO_POOL=ON
  HPX_WITH_ITTNOTIFY=OFF
  HPX_WITH_LOGGING=ON
  HPX_WITH_NETWORKING=ON
  HPX_WITH_PAPI=OFF
  HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
  HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
  HPX_WITH_PARCELPORT_COUNTERS=OFF
  HPX_WITH_PARCELPORT_LCI=OFF
  HPX_WITH_PARCELPORT_LIBFABRIC=OFF
  HPX_WITH_PARCELPORT_MPI=ON
  HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON
  HPX_WITH_PARCELPORT_TCP=OFF
  HPX_WITH_PARCEL_PROFILING=OFF
  HPX_WITH_SANITIZERS=OFF
  HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
  HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
  HPX_WITH_STACKTRACES=ON
  HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
  HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
  HPX_WITH_TESTS_DEBUG_LOG=OFF
  HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
  HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
  HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
  HPX_WITH_THREAD_DEBUG_INFO=OFF
  HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
  HPX_WITH_THREAD_GUARD_PAGE=ON
  HPX_WITH_THREAD_IDLE_RATES=OFF
  HPX_WITH_THREAD_LOCAL_STORAGE=OFF
  HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
  HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
  HPX_WITH_THREAD_STACK_MMAP=ON
  HPX_WITH_THREAD_STEALING_COUNTS=OFF
  HPX_WITH_THREAD_TARGET_ADDRESS=OFF
  HPX_WITH_TIMER_POOL=ON
  HPX_WITH_TUPLE_RVALUE_SWAP=ON
  HPX_WITH_VALGRIND=OFF
  HPX_WITH_VERIFY_LOCKS=OFF
  HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF

Module coroutines:
  HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF

Module datastructures:
  HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=OFF
  HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
  HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
  HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
  HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
  HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
  HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
  HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
  HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.9.1 (AGAS: V3.0), Git: unknown
{boost}: V1.80.0
{build-type}: release
{date}: Oct 27 2023 06:50:20
{platform}: linux
{compiler}: GNU C++ version 12.2.0
{stdlib}: GNU libstdc++ version 20220819
diehlpk commented 8 months ago

@G-071 

 spack spec /2en7plwicmukr2hnmeddi456uvsek6q5 
octotiger@0.10.0%gcc@12.2.0~cuda~fast_fp_contract~ipo+kokkos+kokkos_hpx_kernels~rocm~sycl build_system=cmake build_type=Release griddim=8 hydro_host_tasks=1 monopole_host_tasks=1 multipole_host_tasks=1 simd_extension=SVE simd_library=STD theta_minimum=0.34 arch=linux-rhel8-a64fx
    ^boost@1.80.0%gcc@12.2.0+atomic+chrono~clanglibcpp~container+context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded~stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic context-impl=fcontext cxxstd=17 patches=a440f96 visibility=hidden arch=linux-rhel8-a64fx
        ^bzip2@1.0.6%gcc@12.2.0~debug~pic+shared build_system=generic arch=linux-rhel8-a64fx
        ^zlib@1.2.11%gcc@12.2.0+optimize+pic+shared build_system=makefile arch=linux-rhel8-a64fx
    ^cmake@3.24.3%gcc@12.2.0~doc+ncurses~ownlibs~qt build_system=generic build_type=Release arch=linux-rhel8-a64fx
        ^curl@7.61.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel8-a64fx
        ^expat@2.4.8%gcc@12.2.0+libbsd build_system=autotools arch=linux-rhel8-a64fx
            ^libbsd@0.11.5%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
                ^libmd@1.0.4%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
        ^libarchive@3.5.2%gcc@12.2.0+iconv build_system=autotools compression=bz2lib,lz4,lzma,lzo2,zlib crypto=mbedtls libs=shared,static programs=none xar=expat arch=linux-rhel8-a64fx
            ^libiconv@1.16%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-rhel8-a64fx
            ^lz4@1.8.3%gcc@12.2.0 build_system=makefile libs=shared,static arch=linux-rhel8-a64fx
            ^lzo@2.10%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-rhel8-a64fx
            ^mbedtls@2.28.0%gcc@12.2.0+pic build_system=makefile build_type=Release libs=static arch=linux-rhel8-a64fx
            ^xz@5.2.4%gcc@12.2.0~pic build_system=autotools libs=shared,static arch=linux-rhel8-a64fx
        ^libuv@1.44.1%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
        ^ncurses@6.1%gcc@12.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-a64fx
        ^rhash@1.4.2%gcc@12.2.0 build_system=makefile patches=093518c,3fbfe46 arch=linux-rhel8-a64fx
    ^cppuddle@0.3.0%gcc@12.2.0+allocator_counters+buffer_content_recycling+buffer_recycling~enable_gpu_tests+hpx~ipo build_system=cmake build_type=RelWithDebInfo max_number_gpus=1 number_buffer_buckets=128 arch=linux-rhel8-a64fx
    ^hdf5@1.12.2%gcc@12.2.0~cxx~fortran+hl~ipo~java~mpi+shared+szip+threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo arch=linux-rhel8-a64fx
        ^libaec@1.0.6%gcc@12.2.0~ipo+shared build_system=cmake build_type=RelWithDebInfo arch=linux-rhel8-a64fx
        ^pkgconf@1.4.2%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
    ^hpx@1.9.1%gcc@12.2.0~async_cuda~async_mpi~cuda~disable_async_gpu_futures~examples+generic_coroutines~ipo~rocm~sycl~tools build_system=cmake build_type=RelWithDebInfo cxxstd=17 instrumentation=none malloc=jemalloc max_cpu_count=64 networking=mpi sycl_target_arch=none arch=linux-rhel8-a64fx
        ^asio@1.21.0%gcc@12.2.0~boost_coroutine~boost_regex~separate_compilation build_system=autotools cxxstd=17 arch=linux-rhel8-a64fx
        ^fujitsu-mpi@head%gcc@12.2.0 build_system=generic arch=linux-rhel8-a64fx
        ^git@2.39.1%gcc@12.2.0+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rhel8-a64fx
        ^hwloc@2.2.0%gcc@12.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel8-a64fx
        ^jemalloc@5.2.1%gcc@12.2.0~debug~documentation+fill~prof~stats build_system=autotools jemalloc_prefix=none libs=shared,static arch=linux-rhel8-a64fx
        ^ninja@1.11.1%gcc@12.2.0 build_system=generic arch=linux-rhel8-a64fx
        ^python@3.10.8%fj@4.10.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib build_system=generic patches=0aba3bf,0d98e93,7d40923,f2fd060 arch=linux-rhel8-a64fx
    ^hpx-kokkos@0.4.0%gcc@12.2.0~cuda~ipo~rocm~sycl build_system=cmake build_type=RelWithDebInfo cxxstd=17 future_type=polling arch=linux-rhel8-a64fx
    ^kokkos@develop%gcc@12.2.0+aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation+hpx+hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper build_system=cmake build_type=RelWithDebInfo intel_gpu_arch=none std=17 arch=linux-rhel8-a64fx
    ^silo@4.10.2%gcc@12.2.0+fortran+fpzip+hdf5+hzip~mpi+pic+shared~silex build_system=autotools patches=3a1e831,7b5a1dc,eb2a3a0,fa050e0 arch=linux-rhel8-a64fx
        ^autoconf@2.69%gcc@12.2.0 build_system=autotools patches=7793209 arch=linux-rhel8-a64fx
        ^autoconf-archive@2022.02.11%gcc@12.2.0 build_system=autotools patches=139214f arch=linux-rhel8-a64fx
        ^automake@1.16.1%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
        ^gnuconfig@2021-08-14%gcc@12.2.0 build_system=generic arch=linux-rhel8-a64fx
        ^libtool@2.4.6%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
        ^m4@1.4.18%gcc@12.2.0+sigsegv build_system=autotools patches=3877ab5,fc9b616 arch=linux-rhel8-a64fx
        ^readline@7.0%gcc@12.2.0 build_system=autotools arch=linux-rhel8-a64fx
    ^vc@1.4.1%gcc@12.2.0~ipo build_system=cmake build_type=RelWithDebInfo arch=linux-rhel8-a64fx
        ^virtest@master%gcc@12.2.0~ipo build_system=cmake build_type=RelWithDebInfo arch=linux-rhel8-a64fx
diehlpk commented 7 months ago 
login2$ cat v1309.ini 
n_species=5
max_level=9
#ngrids=2000
problem=dwd
xscale=31.25
core_refine=on
v1309=on
odt=0.01
code_to_g=1.99e+33
code_to_cm=6.96e+10
atomic_mass=4.0026
atomic_mass=1.2969
atomic_mass=4.0026
atomic_mass=1.2969
atomic_mass=1.0079
atomic_number=2.0000
atomic_number=1.0994
atomic_number=2.0000
atomic_number=1.0994
atomic_number=1.0000
X=0.0000
X=0.7068
X=0.0000
X=0.7068
X=1.0000
Z=0.0000
Z=0.0181
Z=0.0000
Z=0.0181
Z=0.0000
restart_filename=X.348.silo
scf_output_frequency=5
rho_floor=1.0e-12
refinement_floor=1.0e-6
theta=0.34
cdisc_detect=0
correct_am_grav=1
correct_am_hydro=0
dt_max=1.0
driving_rate=0.01
driving_time=1.5
stop_step=10
G-071 commented 6 months ago

Re-opening this until @diehlpk can verify that the PR resolved the issue!

diehlpk commented 5 months ago

@G-071 at least on Fugaku, the error is gone.

@JiakunYan can you please test on Perlmutter?

G-071 commented 5 months ago

Hmm, #482 should have no effect on Perlmutter as long as the GPU kernels are used! Or was that on the CPU partition? The issue Jiakun reported should be unrelated to this one.

JiakunYan commented 5 months ago

It was on the GPU partition.