Beluga ROS 2 debian builds fail sporadically

hidmic commented 6 months ago

Bug description

Platform (please complete the following information):

OS: Ubuntu Jammy
Beluga version: 2.0.0

How to reproduce

No way to reproduce. Only observed in the ROS buildfarm, see https://build.ros2.org/job/Hbin_ujv8_uJv8__beluga_amcl__ubuntu_jammy_arm64__binary/1 or https://build.ros2.org/job/Ibin_ujv8_uJv8__beluga_amcl__ubuntu_jammy_arm64__binary/3.

Expected behavior Builds succeed consistently.

Actual behavior Every other builds fails with:

c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.

Additional context

This looks like the compiler getting killed by the OS when close enough to exhaust all system memory. I don't know what resources do ROS buildfarm ARM64 instances have or how many release jobs may be running concurrently at any given time, but at the very least we could check if any of our templates is particularly hard to expand e.g. using -ftemplate-depth=n to search for them by trial and error.

hidmic commented 6 months ago

It's starting to happen on AMD64 too, see https://build.ros2.org/job/Hbin_uJ64__beluga_amcl__ubuntu_jammy_amd64__binary/3.

hidmic commented 6 months ago

I did some minimal digging, reducing -ftemplate-depth (maximum number of template expansions). IIUC gcc defaults it to 900. I can take it down to 100. Compilation breaks at 50, on seemingly innocent code:

Starting >>> beluga_amcl
--- stderr: beluga_amcl                              
In file included from /usr/include/range/v3/functional/concepts.hpp:19,
                 from /usr/include/range/v3/iterator/concepts.hpp:27,
                 from /usr/include/range/v3/range/access.hpp:34,
                 from /usr/include/range/v3/view/all.hpp:22,
                 from /ws/install/include/beluga/beluga/algorithm/raycasting/bresenham.hpp:21,
                 from /ws/install/include/beluga/beluga/algorithm/raycasting.hpp:22,
                 from /ws/install/include/beluga/beluga/sensor/beam_model.hpp:23,
                 from /ws/src/beluga/beluga_amcl/src/amcl_node.cpp:58:
/usr/include/range/v3/functional/invoke.hpp: In instantiation of ‘constexpr ranges::detail::iterator_associated_types_base_<ranges::iota_view<int, int>::cursor, true>::iterator_associated_types_base_(const ranges::detail::iterator_associated_types_base_<ranges::iota_view<int, int>::cursor, true>&)’:
/usr/include/range/v3/functional/invoke.hpp:138:40:   recursively required from ‘constexpr ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >::basic_iterator(const ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&)’
/usr/include/range/v3/functional/invoke.hpp:138:40:   required from ‘constexpr decltype ((F&&)(f)((Args&&(ranges::invoke_fn::operator()::args))...)) ranges::invoke_fn::operator()(F&&, Args&& ...) const [with F = ranges::indirected<<lambda(auto:90&&)> >; Args = {ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}]’
/usr/include/range/v3/functional/concepts.hpp:36:5:   required by substitution of ‘template<class ... As> char (& ranges::invocable__requires_(concepts::detail::tag<As ...>*, decltype (& invocable__requires_test_<As ...>)))[1] [with As = {ranges::indirected<<lambda(auto:90&&)> >, ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}]’
/usr/include/range/v3/functional/concepts.hpp:43:9:   required from ‘constexpr const bool ranges::invocable<ranges::indirected<<lambda(auto:90&&)> >, ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&>’
/usr/include/range/v3/utility/semiregular_box.hpp:245:9:   required by substitution of ‘template<class ... Args, bool CPP_true, typename std::enable_if<(invocable<ranges::indirected<<lambda(auto:90&&)> >, Args ...> && CPP_true), int>::type <anonymous> > constexpr decltype(auto) ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > >::operator()<Args ..., CPP_true, <anonymous> >(Args&& ...) && [with Args = {ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}; bool CPP_true = true; typename std::enable_if<(invocable<ranges::indirected<<lambda(auto:90&&)> >, Args ...> && CPP_true), int>::type <anonymous> = <missing>]’
/usr/include/range/v3/functional/invoke.hpp:139:34:   required by substitution of ‘template<class F, class ... Args> constexpr decltype ((F&&)(f)((Args&&(ranges::invoke_fn::operator()::args))...)) ranges::invoke_fn::operator()(F&&, Args&& ...) const [with F = ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > >&; Args = {ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}]’
/usr/include/range/v3/functional/reference_wrapper.hpp:111:24:   required by substitution of ‘template<class ... Args> constexpr decltype (ranges::invoke(static_cast<ranges::reference_wrapper<ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > > >::reference>((*((const ranges::reference_wrapper<ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > > >*)this)->ranges::detail::reference_wrapper_<ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > > >::t_)), static_cast<Args&&>(ranges::reference_wrapper<T>::operator()::args) ...)) ranges::reference_wrapper<ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > > >::operator()<Args ...>(Args&& ...) const [with Args = {ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}]’
/usr/include/range/v3/functional/invoke.hpp:139:34:   required by substitution of ‘template<class F, class ... Args> constexpr decltype ((F&&)(f)((Args&&(ranges::invoke_fn::operator()::args))...)) ranges::invoke_fn::operator()(F&&, Args&& ...) const [with F = const ranges::reference_wrapper<ranges::semiregular_box<ranges::indirected<<lambda(auto:90&&)> > > >&; Args = {ranges::basic_iterator<ranges::adaptor_cursor<ranges::basic_iterator<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >::cursor>, ranges::remove_if_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, ranges::logical_negate<beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> > >::adaptor> >&}]’
/usr/include/range/v3/view/transform.hpp:136:18:   required from ‘struct ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >::adaptor<false>’
/usr/include/range/v3/detail/range_access.hpp:96:31:   required from ‘static constexpr decltype (rng.begin_adaptor()) ranges::range_access::begin_adaptor(Rng&) [with Rng = ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >]’
/usr/include/range/v3/view/adaptor.hpp:40:40:   required by substitution of ‘template<class D> using adaptor_cursor_t = ranges::adaptor_cursor<typename std::decay<decltype (declval<typename std::decay<decltype (ranges::range_access::begin_adaptor(declval<Derived&>()))>::type>().begin(declval<Derived&>()))>::type, typename std::decay<decltype (ranges::range_access::begin_adaptor(declval<Derived&>()))>::type> [with D = ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >]’
/usr/include/range/v3/view/adaptor.hpp:504:46:   required by substitution of ‘template<class D> static constexpr ranges::adaptor_cursor_t<D> ranges::view_adaptor<ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >, ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::finite>::begin_cursor_<D>(D&) [with D = ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >]’
/usr/include/range/v3/view/adaptor.hpp:517:52:   required by substitution of ‘template<class D, bool CPP_true, typename std::enable_if<(same_as<D, ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > > > && CPP_true), int>::type <anonymous> > constexpr decltype (ranges::view_adaptor<ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >, ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::finite>::begin_cursor_<D>(declval<D&>())) ranges::view_adaptor<ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >, ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::finite>::begin_cursor<D, CPP_true, <anonymous> >() [with D = ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >; bool CPP_true = true; typename std::enable_if<(same_as<D, ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > > > && CPP_true), int>::type <anonymous> = 0]’
/usr/include/range/v3/detail/range_access.hpp:85:31:   required by substitution of ‘template<class Rng> static constexpr decltype (rng.begin_cursor()) ranges::range_access::begin_cursor(Rng&) [with Rng = const ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >]’
/usr/include/range/v3/view/facade.hpp:39:39:   required by substitution of ‘template<class Derived> using facade_iterator_t = ranges::basic_iterator<typename std::decay<decltype (ranges::range_access::begin_cursor(declval<Derived&>()))>::type> [with Derived = const ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >]’
/usr/include/range/v3/view/facade.hpp:106:24:   required by substitution of ‘template<class D, bool CPP_true, typename std::enable_if<(same_as<D, ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > > > && CPP_true), int>::type <anonymous> > constexpr ranges::detail::facade_iterator_t<const D> ranges::view_facade<ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >, ranges::finite>::begin<D, CPP_true, <anonymous> >() const [with D = ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > >; bool CPP_true = true; typename std::enable_if<(same_as<D, ranges::iter_transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, ranges::indirected<<lambda(auto:90&&)> > > > && CPP_true), int>::type <anonymous> = 0]’
/usr/include/range/v3/range/access.hpp:86:9:   required by substitution of ‘template<class ... As> char (& ranges::_begin_::has_member_begin__requires_(concepts::detail::tag<As ...>*, decltype (& has_member_begin__requires_test_<As ...>)))[1] [with As = {ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> >&}]’
/usr/include/range/v3/range/access.hpp:93:13:   required from ‘constexpr const bool ranges::_begin_::has_member_begin<ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> >&>’
/usr/include/range/v3/range/access.hpp:133:13:   required by substitution of ‘template<class R, bool CPP_true, typename std::enable_if<(_borrowed_range<R> && CPP_true), int>::type <anonymous>, typename std::enable_if<((has_member_begin<R> || has_non_member_begin<R>) && CPP_true), int>::type <anonymous> > constexpr auto ranges::_begin_::fn::operator()(R&&) const [with R = ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> >&; bool CPP_true = true; typename std::enable_if<(_borrowed_range<R> && CPP_true), int>::type <anonymous> = <missing>; typename std::enable_if<((has_member_begin<R> || has_non_member_begin<R>) && CPP_true), int>::type <anonymous> = <missing>]’
/usr/include/range/v3/range/concepts.hpp:56:5:   required by substitution of ‘template<class ... As> char (& ranges::_range__requires_(concepts::detail::tag<As ...>*, decltype (& _range__requires_test_<As ...>)))[1] [with As = {ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> >}]’
/usr/include/range/v3/range/concepts.hpp:64:9:   required from ‘constexpr const bool ranges::range<ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> > >’
/usr/include/range/v3/range/concepts.hpp:210:9:   required from ‘constexpr const bool ranges::viewable_range<ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> > >’
/usr/include/range/v3/view/view.hpp:106:13:   required by substitution of ‘template<class Rng, class ViewFn, bool CPP_true, typename std::enable_if<(viewable_range<Rng> && CPP_true), int>::type <anonymous>, typename std::enable_if<(invocable_view_closure<ViewFn, Rng> && CPP_true), int>::type <anonymous> > constexpr auto ranges::views::view_closure_base_ns::operator|(Rng&&, ranges::views::view_closure<ViewFn>) [with Rng = ranges::transform_view<ranges::filter_view<ranges::cache1_view<ranges::zip_view<ranges::detail::index_view<long unsigned int, long int>, ranges::iota_view<int, int> > >, beluga::views::detail::take_evenly_fn::operator()<ranges::iota_view<int, int> >(ranges::iota_view<int, int>&&, std::size_t) const::<lambda(const auto:91&)> >, <lambda(auto:90&&)> >; ViewFn = ranges::detail::bind_back_fn_<ranges::views::transform_base_fn, beluga_ros::LaserScan::angles() const::<lambda(int)> >; bool CPP_true = true; typename std::enable_if<(viewable_range<Rng> && CPP_true), int>::type <anonymous> = <missing>; typename std::enable_if<(invocable_view_closure<ViewFn, Rng> && CPP_true), int>::type <anonymous> = <missing>]’
/ws/install/include/beluga_ros/beluga_ros/laser_scan.hpp:71:13:   required from here
/usr/include/range/v3/functional/invoke.hpp:138:40: fatal error: template instantiation depth exceeds maximum of 50 (use ‘-ftemplate-depth=’ to increase the maximum)
  138 |             noexcept(noexcept(((F&&) f)((Args&&) args...)))
      |                               ~~~~~~~~~^~~~~~~~~~~~~~~~~~
compilation terminated.

If buildfarm machines are already loaded, I can see them crashing on such deep template compilation. I don't see a way to improve it other than making less aggressive use of ranges :cry: FYI @nahueespinosa.

hidmic commented 6 months ago

I've so far failed to reduce template nesting. I think we'll have to go the explicit instantiation route instead, as you suggested @nahueespinosa, and hope we can cut enough bloat that way.

glpuga commented 4 months ago

Another datapoint here.

The latest versions of Beluga seem ot have a lot of memory, far more than older versions (start of year). Updating the kobuki stack's beluga version to the latest from a version from last January turned it into unbuildable in a 8GB raspberry 5 unless the sequential colcon executor is used. Peak memory usage happens when bulding beluga_amcl, at around 5-6GB.

The same issue happened in a 16GB x64 desktop shortly after. Again, peak memory usage happened on beluga_amcl, with a slightly lower peak memory usage than aarch64, but still enough to crash the system when built concurrently with other packages.

hidmic commented 4 months ago

@glpuga it's the templates. clang does a better job than gcc, memory consumption-wise. My attempts to optimize it failed. Explicit template instantiation may help.

hidmic commented 4 months ago

Got another death on the farm https://build.ros2.org/job/Jbin_uN64__beluga_amcl__ubuntu_noble_amd64__binary. We gotta fix this.

glpuga commented 4 months ago

I don't think this is a situation that deteriorated over time from start of year, really, but now we have two independent targets in beluga_amcl that get built at the same time, doubling the issue.

A stopgap solution might be to temporarily daisy chain those two targets with cmake so they don't get built at the same time, while we look for a better solution.

glpuga commented 4 months ago

A current thread with a lot of information about this: https://discourse.ros.org/t/openvdb-compile-size/38790/4

Each build agent is a compute-optimized “xlarge” instance in EC2 spread between c5.xlarge, c6i.xlarge, or m5.xlarge which all have 4vCPUs and 8GiB of memory.

Each individual package build is expected to use 1/4th of those resources: 1vCPU and 2GiB of RAM. These are soft limits that are not enforced, but since agents run up to four jobs at a time, packages using more than the available memory will be subject to pressure from the OS.

glpuga commented 4 months ago

A stopgap solution might be to temporarily daisy chain those two targets with cmake so they don't get built at the same time, while we look for a better solution.

It's interesting that "the buildfarm by default already uses MAKEFLAGS=-j1", which means my previous stopgap won't make any difference.

hidmic commented 4 months ago

I've been following that thread. It basically confirms our suspicion: the allocation build jobs get on buildfarm VMs is small-ish for today's standards.

We need the compiler to use less memory.