SIGABRT: "Not enough memory in the buffer stream" with rqt

yashi commented 9 months ago

Hi,

Does anyone successfully run rqt with micro-ros Agent?

On my PC, the agent dies when I run both rqt and micro-ROS. That is, the Agent doesn't die when there are only:

the agent and micro-ROS nodes on MCU
the agent and rqt without plugins
the agent and rqt with node graph plugin loaded

But as soon as I run the three, Agent, micro-ROS nodes, and rqt with the "Node Graph", Agent aborts with the following message:

terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream
[ros2run]: Aborted

Full backtrace is down below.

Issue template

Hardware description: PC
- CPU: AMD Ryzen 9 5900X 12-Core Processor
- Memory: Total 62Gi, Used 10Gi, Avaiable 51Gi
OS: Debian GNU/Linux Sid
ROS
- Distro: Humble
- Installation type: my own build
micro-ROS Agent version: 3.0.5
rqt version: 1.3.3

Steps to reproduce the issue

Run micro-ROS Agent on a PC
Run rqt on the same PC
Run micro-ROS on a MCU
The agent dies with SIGABRT

Expected behavior

Agent doesn't die.

Actual behavior

Agent dies with SIGABRT.

Additional information

Backtrace

``` (gdb) bt #0 __pthread_kill_implementation (threadid=, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 #1 0x00007ffff5aa815f in __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78 #2 0x00007ffff5a5a472 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff5a444b2 in __GI_abort () at ./stdlib/abort.c:79 #4 0x00007ffff5ca09db in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff5caffaa in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007ffff5cb0015 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007ffff5cb0268 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x00007ffff7d843ec in eprosima::fastcdr::Cdr::deserializeArray (this=0x7fffd6ffaaa0, char_t=0x7fff38002900 "", numElements=24) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-CDR/src/cpp/Cdr.cpp:2076 #9 0x00007ffff7a5a7dd in eprosima::fastcdr::Cdr::deserializeArray (this=0x7fffd6ffaaa0, octet_t=0x7fff38002900 "", numElements=24) at /home/yashi/work/yoshida/humble/install/include/fastcdr/Cdr.h:2712 #10 0x00007ffff7d58185 in eprosima::fastcdr::Cdr::deserialize (this=0x7fffd6ffaaa0, array_t=...) at /home/yashi/work/yoshida/humble/install/include/fastcdr/Cdr.h:2528 #11 0x00007ffff7d580f9 in eprosima::fastcdr::Cdr::operator>> (this=0x7fffd6ffaaa0, array_t=...) at /home/yashi/work/yoshida/humble/install/include/fastcdr/Cdr.h:767 #12 0x00007ffff7d57df6 in rmw_dds_common::msg::typesupport_fastrtps_cpp::cdr_deserialize (cdr=..., ros_message=...) at /home/yashi/work/yoshida/humble/build/rmw_dds_common/rosidl_typesupport_fastrtps_cpp/rmw_dds_common/msg/detail/dds_fastrtps/gid__type_support.cpp:50 #13 0x00007ffff7d58393 in rmw_dds_common::msg::typesupport_fastrtps_cpp::cdr_deserialize (cdr=..., ros_message=...) at /home/yashi/work/yoshida/humble/build/rmw_dds_common/rosidl_typesupport_fastrtps_cpp/rmw_dds_common/msg/detail/dds_fastrtps/node_entities_info__type_support.cpp:107 #14 0x00007ffff7d59ca8 in rmw_dds_common::msg::typesupport_fastrtps_cpp::cdr_deserialize (cdr=..., ros_message=...) at /home/yashi/work/yoshida/humble/build/rmw_dds_common/rosidl_typesupport_fastrtps_cpp/rmw_dds_common/msg/detail/dds_fastrtps/participant_entities_info__type_support.cpp:117 #15 0x00007ffff7d59f2d in rmw_dds_common::msg::typesupport_fastrtps_cpp::_ParticipantEntitiesInfo__cdr_deserialize (cdr=..., untyped_ros_message=0x7fffd6ffaf00) at /home/yashi/work/yoshida/humble/build/rmw_dds_common/rosidl_typesupport_fastrtps_cpp/rmw_dds_common/msg/detail/dds_fastrtps/participant_entities_info__type_support.cpp:234 #16 0x00005555555f8456 in uros::agent::graph_manager::ParticipantEntitiesInfoTypeSupport::deserialize (this=0x7fffe809faf0, payload=0x7fffe8182910, data=0x7fffd6ffaf00) at /home/yashi/work/yoshida/agent-ws/src/micro-ros-agent/micro_ros_agent/src/agent/graph_manager/graph_typesupport.cpp:83 #17 0x00007ffff6afb776 in eprosima::fastdds::dds::detail::ReadTakeCommand::deserialize_sample (this=0x7fffd6ffac40, change=0x7fffe81828e0) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl/ReadTakeCommand.hpp:355 #18 0x00007ffff6afb64e in eprosima::fastdds::dds::detail::ReadTakeCommand::add_sample (this=0x7fffd6ffac40, item=@0x7fffe81a6eb0: 0x7fffe81828e0, deserialization_error=@0x7fffd6ffabc6: false) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl/ReadTakeCommand.hpp:326 #19 0x00007ffff6afb01e in eprosima::fastdds::dds::detail::ReadTakeCommand::add_instance (this=0x7fffd6ffac40, take_samples=true) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl/ReadTakeCommand.hpp:147 #20 0x00007ffff6af1b07 in eprosima::fastdds::dds::DataReaderImpl::read_or_take_next_sample (this=0x7fffe8180530, data=0x7fffd6ffaf00, info=0x7fffd6ffae70, should_take=true) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl.cpp:679 #21 0x00007ffff6af1c29 in eprosima::fastdds::dds::DataReaderImpl::take_next_sample (this=0x7fffe8180530, data=0x7fffd6ffaf00, info=0x7fffd6ffae70) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl.cpp:701 #22 0x00007ffff6aecd71 in eprosima::fastdds::dds::DataReader::take_next_sample (this=0x7fffe81a1ea0, data=0x7fffd6ffaf00, info=0x7fffd6ffae70) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReader.cpp:244 #23 0x00005555555e02cc in uros::agent::graph_manager::GraphManager::update_node_entities_info (this=0x7fffe809f4e0) at /home/yashi/work/yoshida/agent-ws/src/micro-ros-agent/micro_ros_agent/src/agent/graph_manager/graph_manager.cpp:547 #24 0x00005555555e1146 in uros::agent::graph_manager::GraphManager::DatareaderListener::on_data_available (this=0x7fffe809ea10) at /home/yashi/work/yoshida/agent-ws/src/micro-ros-agent/micro_ros_agent/src/agent/graph_manager/graph_manager.cpp:761 #25 0x00007ffff6af237e in eprosima::fastdds::dds::DataReaderImpl::InnerDataReaderListener::onNewCacheChangeAdded (this=0x7fffe8180d90, change_in=0x7fffe81828e0) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/fastdds/subscriber/DataReaderImpl.cpp:844 #26 0x00007ffff69fb62c in eprosima::fastrtps::rtps::StatefulReader::NotifyChanges (this=0x7fffe81b75c0, prox=0x7fffcc004990) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/reader/StatefulReader.cpp:1074 #27 0x00007ffff69fa7e8 in eprosima::fastrtps::rtps::StatefulReader::processHeartbeatMsg (this=0x7fffe81b75c0, writerGUID=..., hbCount=4, firstSN=..., lastSN=..., finalFlag=false, livelinessFlag=false) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/reader/StatefulReader.cpp:761 #28 0x00007ffff6a21f43 in operator() (__closure=0x7fffd6ffb650, reader=0x7fffe81b75c0) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/messages/MessageReceiver.cpp:1081 #29 0x00007ffff6a237aa in eprosima::fastrtps::rtps::MessageReceiver::findAllReaders >(const eprosima::fastrtps::rtps::EntityId_t &, const struct {...} &) const (this=0x7fffe80dfc40, readerID=..., callback=...) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/messages/MessageReceiver.cpp:664 #30 0x00007ffff6a22586 in eprosima::fastrtps::rtps::MessageReceiver::proc_Submsg_Heartbeat (this=0x7fffe80dfc40, msg=0x7fffd6ffb960, smh=0x7fffd6ffb858) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/messages/MessageReceiver.cpp:1078 #31 0x00007ffff6a1f826 in eprosima::fastrtps::rtps::MessageReceiver::processCDRMsg (this=0x7fffe80dfc40, source_locator=..., reception_locator=..., msg=0x7fffd6ffb960) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/messages/MessageReceiver.cpp:479 #32 0x00007ffff6a346d3 in eprosima::fastrtps::rtps::ReceiverResource::OnDataReceived (this=0x7fffe80a8d50, data=0x7fffe01981b0 "RTPS\002\002\001\017\001\017\v<\t\030v'", size=52, localLocator=..., remoteLocator=...) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/network/ReceiverResource.cpp:121 #33 0x00007ffff6e633b7 in eprosima::fastdds::rtps::SharedMemChannelResource::perform_listen_operation (this=0x7fffe80a8110, input_locator=...) at /home/yashi/work/yoshida/humble/src/eProsima/Fast-DDS/src/cpp/rtps/transport/shared_mem/SharedMemChannelResource.hpp:146 #34 0x00007ffff6e7ad50 in std::__invoke_impl ( __f=@0x7fffe80dfa88: (void (eprosima::fastdds::rtps::SharedMemChannelResource::*)(eprosima::fastdds::rtps::SharedMemChannelResource * const, eprosima::fastrtps::rtps::Locator_t)) 0x7ffff6e6318e , __t=@0x7fffe80dfa80: 0x7fffe80a8110) at /usr/include/c++/12/bits/invoke.h:74 #35 0x00007ffff6e7a946 in std::__invoke ( __fn=@0x7fffe80dfa88: (void (eprosima::fastdds::rtps::SharedMemChannelResource::*)(eprosima::fastdds::rtps::SharedMemChannelResource * const, eprosima::fastrtps::rtps::Locator_t)) 0x7ffff6e6318e ) at /usr/include/c++/12/bits/invoke.h:96 #36 0x00007ffff6e7a747 in std::thread::_Invoker >::_M_invoke<0ul, 1ul, 2ul> (this=0x7fffe80dfa68) at /usr/include/c++/12/bits/std_thread.h:279 #37 0x00007ffff6e7a300 in std::thread::_Invoker >::operator() (this=0x7fffe80dfa68) at /usr/include/c++/12/bits/std_thread.h:286 #38 0x00007ffff6e79ce8 in std::thread::_State_impl > >::_M_run (this=0x7fffe80dfa60) at /usr/include/c++/12/bits/std_thread.h:231 #39 0x00007ffff5cdc433 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #40 0x00007ffff5aa63ec in start_thread (arg=) at ./nptl/pthread_create.c:444 #41 0x00007ffff5b26a2c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) ```

pablogs9 commented 9 months ago

Which distro of ROS 2 are you running?

yashi commented 9 months ago

Humble.

pablogs9 commented 9 months ago

Could you confirm that using iron version of the agent this issue does not happens?

In my scenario, I can replicate your issue in humble but if I just run the iron agent, it does not happen.

If you confirm that, I will take a look at humble.

EDIT:

This seems to be an iron <-> humble incompatibility.

In my previous scenario, I had: micro-ROS humble + Agent Humble + RQT Iron and it failed.

If I move to micro-ROS humble + Agent Humble + RQT Humble it does not fail.

This makes sense since ros introspection messages may be incompatible and the micro-ROS Agent is failing at the graph management level. Could you ensure that your custom installation is a pure humble distro?

yashi commented 9 months ago

You are right. My RQT was somehow built against iron. I apologize and thank you for your time.

	micro-ROS	Agent	RQT
NG	Humble	Humble	v1.3.3 built against Iron
OK	Humble	Humble	v1.3.3 built against Humble
OK	Humble	Iron	v1.3.3 built against Iron

eProsima / Micro-XRCE-DDS-Agent