Open CihatAltiparmak opened 4 months ago
CihatAltiparmak
commented
4 months ago In addition all of this, i want to say that i have refactored the perception pipeline tutorial in moveit2_tutorials in order to see the effects of rmw_zenoh againts perception pipeline scenario.
CihatAltiparmak
commented
4 months ago Hello all,
I want to share the first insights about MoveIt + rmw_zenoh combination. Firstly, when i executed MoveIt demo with rmw_zenoh, even though gripper was loaded and usable in rviz2, manipulator was not loaded and invisible in rviz2 like below image. In addition, planning library was not loaded. As result of this, it also doesn't work planning and executing pipeline.
I started to inspect ros logs once demo was crashed. In ros logs, i saw that problem is based on controller spawner node, especially joint_state_broadcaster. And i also caught some bugs which it is suggested to report it in rmw_zenoh. This errors are below. However i should say that i also observed MoveIt demo + rmw_zenoh is executed without problem. which works so rarely. It generally gives this bellow errors in logs.
I also ran test cases of ros2_control and ros2_controllers using this dockerfile and then the some test cases are crashed. When i ran test cases several times, about 1-2 failures occured after testing. Shortly it is sometimes failed tricycle_controller + joint_trajectory_controller and it is sometimes failed just joint_state_publisher and so on. The common point of this failures was joint_trajectory_controller. Moreover, all failures in joint_trajectory_publisher was due to below segmentation fault.
[ OK ] OnlyEffortTrajectoryControllers/TestTrajectoryActionsTestParameterized.test_allow_nonzero_velocity_at_trajectory_end_true/0 (2010 ms)
[ RUN ] OnlyEffortTrajectoryControllers/TestTrajectoryActionsTestParameterized.test_allow_nonzero_velocity_at_trajectory_end_true/1
malloc(): unaligned tcache chunk detected
Stack trace (most recent call last):
#27 Object "", at 0xffffffffffffffff, in
#26 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f58ac324, in _start
#25 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada2c4e3f, in __libc_start_main
#24 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada2c4d8f, in
#23 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f58abd31, in main
#22 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f592177e, in testing::UnitTest::Run()
#21 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f592155a, in testing::internal::UnitTestImpl::RunAllTests()
#20 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f5920fdc, in testing::TestSuite::Run()
#19 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f5920ef4, in testing::TestInfo::Run()
#18 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f5920d20, in testing::Test::Run()
#17 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f592d3f0, in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)
#16 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f58db393, in TestTrajectoryActionsTestParameterized::SetUp()
#15 Object "/ws/build/joint_trajectory_controller/test_trajectory_actions", at 0x55d0f58daa4c, in test_trajectory_controllers::TrajectoryControllerTest::SetUp()
#14 Object "/opt/ros/iron/lib/librclcpp.so", at 0x70aada8dc827, in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&)
#13 Object "/opt/ros/iron/lib/librclcpp.so", at 0x70aada8db027, in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&)
#12 Object "/opt/ros/iron/lib/librclcpp.so", at 0x70aada8dabe2, in rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_s const&, bool, bool, std::shared_ptr<rclcpp::CallbackGroup>)
#11 Object "/opt/ros/iron/lib/librcl.so", at 0x70aada78f0da, in rcl_node_init
#10 Object "/opt/ros/iron/lib/librcl.so", at 0x70aada78e939, in
#9 Object "/opt/ros/iron/lib/librcl.so", at 0x70aada78e045, in rcl_publisher_init
#8 Object "/ws/install/rmw_zenoh_cpp/lib/librmw_zenoh_cpp.so", at 0x70aad8c20fe5, in rmw_create_publisher
#7 Object "/ws/install/rmw_zenoh_cpp/lib/librmw_zenoh_cpp.so", at 0x70aad8c1332b, in std::_Deque_base<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::_M_initialize_map(unsigned long) [clone .constprop.0]
#6 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x70aada59498b, in operator new(unsigned long)
#5 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada3403db, in malloc
#4 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada33bcfb, in
#3 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada324675, in
#2 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada2c37f2, in abort
#1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada2dd475, in raise
#0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x70aada3319fc, in pthread_kill
Aborted (Signal sent by tkill() 4313 0)I continued to search this fault and its cause. And at the moment i just can say that segmentation fault occurs at this line of one of joint_trajectory_controller package's test files
Furthermore, i run only failed test case of joint_trajectory_controller with gdb debug and result is below.
[Thread 0x7904777fe640 (LWP 20062) exited]
[ OK ] OnlyEffortTrajectoryControllers/TestTrajectoryActionsTestParameterized.test_allow_nonzero_velocity_at_trajectory_end_true/0 (2017 ms)
[ RUN ] OnlyEffortTrajectoryControllers/TestTrajectoryActionsTestParameterized.test_allow_nonzero_velocity_at_trajectory_end_true/1
malloc(): unaligned tcache chunk detected
Thread 1 "test_trajectory" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=133060925205056) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) backtrace
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=133060925205056) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=133060925205056) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=133060925205056, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007904a997b476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007904a99617f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007904a99c2676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7904a9b14b77 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007904a99d9cfc in malloc_printerr (str=str@entry=0x7904a9b17d20 "malloc(): unaligned tcache chunk detected") at ./malloc/malloc.c:5664
#7 0x00007904a99de3dc in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3195
#8 __GI___libc_malloc (bytes=512) at ./malloc/malloc.c:3313
#9 0x00007904a9c3298c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007904a8229b58 in __gnu_cxx::new_allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > >::allocate (
this=0x6105b6da66d0, __n=64) at /usr/include/c++/11/ext/new_allocator.h:127
#11 0x00007904a822946f in std::allocator_traits<std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::allocate (
__a=..., __n=64) at /usr/include/c++/11/bits/alloc_traits.h:464
#12 0x00007904a8228dfe in std::_Deque_base<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::_M_allocate_node (this=0x6105b6da66d0) at /usr/include/c++/11/bits/stl_deque.h:562
#13 0x00007904a829718b in std::_Deque_base<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::_M_create_nodes (this=0x6105b6da66d0, __nstart=0x6105b6d31f88, __nfinish=0x6105b6d31f90)
at /usr/include/c++/11/bits/stl_deque.h:663
#14 0x00007904a82959cc in std::_Deque_base<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::_M_initialize_map (this=0x6105b6da66d0, __num_elements=0) at /usr/include/c++/11/bits/stl_deque.h:637
#15 0x00007904a82947f4 in std::_Deque_base<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::_Deque_base (this=0x6105b6da66d0) at /usr/include/c++/11/bits/stl_deque.h:439
#16 0x00007904a8293772 in std::deque<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> >, std::allocator<std::unique_ptr<rmw_zenoh_cpp::rmw_zenoh_event_status_t, std::default_delete<rmw_zenoh_cpp::rmw_zenoh_event_status_t> > > >::deque (this=0x6105b6da66d0) at /usr/include/c++/11/bits/stl_deque.h:834
#17 0x00007904a8293b11 in rmw_zenoh_cpp::EventsManager::EventsManager (this=0x6105b6da64e0) at /ws/src/rmw_zenoh/rmw_zenoh_cpp/src/detail/event.hpp:111
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x00007904a8293de9 in rmw_zenoh_cpp::rmw_publisher_data_t::rmw_publisher_data_t (this=0x6105b6da63e0) at /ws/src/rmw_zenoh/rmw_zenoh_cpp/src/detail/rmw_data_types.hpp:88
#19 0x00007904a827e5b7 in rmw_create_publisher (node=0x6105b7204720, type_supports=0x7904a95bdb80, topic_name=0x6105b6e442b0 "/rosout", qos_profile=0x7fff652e1eb0, publisher_options=0x7fff652e1f30)
at /ws/src/rmw_zenoh/rmw_zenoh_cpp/src/rmw_zenoh.cpp:504
#20 0x00007904a9e2c046 in rcl_publisher_init (publisher=publisher@entry=0x7fff652e1ea8, node=node@entry=0x6105b710d3c0, type_support=type_support@entry=0x7904a95bdb80,
topic_name=topic_name@entry=0x7904a9e3c950 "/rosout", options=options@entry=0x7fff652e1eb0) at ./src/rcl/publisher.c:111
#21 0x00007904a9e2c93a in rcl_logging_rosout_init_publisher_for_node (node=node@entry=0x6105b710d3c0) at ./src/rcl/logging_rosout.c:275
#22 0x00007904a9e2cb12 in rcl_logging_rosout_init_publisher_for_node (node=node@entry=0x6105b710d3c0) at ./src/rcl/logging_rosout.c:235
#23 0x00007904a9e2d0db in rcl_node_init (node=0x6105b710d3c0, name=<optimized out>, namespace_=<optimized out>, context=0x6105b6afedb0, options=0x6105b6e00120) at ./src/rcl/node.c:297
#24 0x00007904a9f78be3 in rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_s const&, bool, bool, std::shared_ptr<rclcpp::CallbackGroup>) () from /opt/ros/iron/lib/librclcpp.so
#25 0x00007904a9f79028 in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) () from /opt/ros/iron/lib/librclcpp.so
#26 0x00007904a9f7a828 in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) () from /opt/ros/iron/lib/librclcpp.so
#27 0x00006105b4ce597b in __gnu_cxx::new_allocator<rclcpp::Node>::construct<rclcpp::Node, char const (&) [22]> (this=0x7fff652e8fcf, __p=0x6105b71d5d30) at /usr/include/c++/11/ext/new_allocator.h:162
#28 0x00006105b4ce09b0 in std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [22]> (__a=..., __p=0x6105b71d5d30) at /usr/include/c++/11/bits/alloc_traits.h:516
#29 0x00006105b4cda9b2 in std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [22]> (this=0x6105b71d5d20, __a=...)
at /usr/include/c++/11/bits/shared_ptr_base.h:519
#30 0x00006105b4cd1ba4 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [22]> (this=0x7fff652e9178, __p=@0x7fff652e9170: 0x0,
__a=...) at /usr/include/c++/11/bits/shared_ptr_base.h:650
#31 0x00006105b4cc832c in std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [22]> (this=0x7fff652e9170, __tag=...)
at /usr/include/c++/11/bits/shared_ptr_base.h:1342
#32 0x00006105b4cbce55 in std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [22]> (this=0x7fff652e9170, __tag=...) at /usr/include/c++/11/bits/shared_ptr.h:409
#33 0x00006105b4caf263 in std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [22]> (__a=...) at /usr/include/c++/11/bits/shared_ptr.h:863
#34 0x00006105b4ca25e5 in std::make_shared<rclcpp::Node, char const (&) [22]> () at /usr/include/c++/11/bits/shared_ptr.h:879
#35 0x00006105b4c925dc in test_trajectory_controllers::TrajectoryControllerTest::SetUp (this=0x6105b6b84ac0)
at /ws/src/ros-controls/ros2_controllers/joint_trajectory_controller/test/test_trajectory_controller_utils.hpp:207
--Type <RET> for more, q to quit, c to continue without paging--
#36 0x00006105b4c96539 in TestTrajectoryActions::SetUp (this=0x6105b6b84ac0) at /ws/src/ros-controls/ros2_controllers/joint_trajectory_controller/test/test_trajectory_actions.cpp:62
#37 0x00006105b4c97cec in TestTrajectoryActionsTestParameterized::SetUp (this=0x6105b6b84ac0) at /ws/src/ros-controls/ros2_controllers/joint_trajectory_controller/test/test_trajectory_actions.cpp:211
#38 0x00006105b4d3e32b in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x6105b6b84ac0, method=&virtual testing::Test::SetUp(), location=0x6105b4d6afe3 "SetUp()")
at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2433
#39 0x00006105b4d379b5 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x6105b6b84ac0, method=&virtual testing::Test::SetUp(), location=0x6105b4d6afe3 "SetUp()")
at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2469
#40 0x00006105b4d1210a in testing::Test::Run (this=0x6105b6b84ac0) at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2503
#41 0x00006105b4d12bc9 in testing::TestInfo::Run (this=0x6105b6b29350) at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2684
#42 0x00006105b4d1332a in testing::TestSuite::Run (this=0x6105b6b177d0) at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2816
#43 0x00006105b4d1f949 in testing::internal::UnitTestImpl::RunAllTests (this=0x6105b6b12b90) at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:5338
#44 0x00006105b4d3f3b2 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x6105b6b12b90,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x6105b4d1f552 <testing::internal::UnitTestImpl::RunAllTests()>,
location=0x6105b4d6b9a0 "auxiliary test code (environments or event listeners)") at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2433
#45 0x00006105b4d38c07 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x6105b6b12b90,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x6105b4d1f552 <testing::internal::UnitTestImpl::RunAllTests()>,
location=0x6105b4d6b9a0 "auxiliary test code (environments or event listeners)") at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:2469
#46 0x00006105b4d1e12b in testing::UnitTest::Run (this=0x6105b4de6b00 <testing::UnitTest::GetInstance()::instance>) at /opt/ros/iron/src/gtest_vendor/src/gtest.cc:4925
#47 0x00006105b4d098e4 in RUN_ALL_TESTS () at /opt/ros/iron/src/gtest_vendor/include/gtest/gtest.h:2473
#48 0x00006105b4d0985d in main (argc=1, argv=0x7fff652e9958) at /opt/ros/iron/src/gmock_vendor/src/gmock_main.cc:63
(gdb) Finally i want to add that, i also ran MoveIt test cases with both rmw_zenoh_cpp and rmw_fastrtps_cpp middlewares using this dockerfile. It is surprising that the execution with rmw_zenoh gives less failures in test cases than the execution with rmw_fastrtps. Here is results.
the execution of moveit test cases with rmw_zenoh
root@9a64dbcea29c:/ws# colcon test-result
build/moveit_kinematics/Testing/20240528-1747/Test.xml: 4 tests, 0 errors, 4 failures, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_fanuc-kdl-singular.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_fanuc-kdl.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_panda-kdl-singular.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_panda-kdl.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_ros_tests/Testing/20240528-1748/Test.xml: 1 test, 0 errors, 1 failure, 0 skipped
build/moveit_ros_tests/test_results/moveit_ros_tests/launch_move_group_api.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
build/moveit_servo/Testing/20240528-1748/Test.xml: 3 tests, 0 errors, 2 failures, 0 skipped
build/moveit_servo/test_results/moveit_servo/tests_launch_servo_cpp_integration.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_servo/test_results/moveit_servo/tests_launch_servo_ros_integration.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
Summary: 683 tests, 1 error, 13 failures, 59 skippedthe execution of moveit test cases with rmw_fastrtps
root@9a64dbcea29c:/ws# colcon test-result
build/moveit_core/Testing/20240528-1750/Test.xml: 32 tests, 0 errors, 4 failures, 2 skipped
build/moveit_core/test_results/moveit_core/test-node_logging.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/Testing/20240528-1751/Test.xml: 4 tests, 0 errors, 4 failures, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_fanuc-kdl-singular.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_fanuc-kdl.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_panda-kdl-singular.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_kinematics/test_results/moveit_kinematics/test_launch_panda-kdl.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_planners_ompl/Testing/20240528-1751/Test.xml: 6 tests, 0 errors, 1 failure, 0 skipped
build/moveit_ros_planning/Testing/20240528-1751/Test.xml: 5 tests, 0 errors, 1 failure, 0 skipped
build/moveit_servo/Testing/20240528-1751/Test.xml: 3 tests, 0 errors, 2 failures, 0 skipped
build/moveit_servo/test_results/moveit_servo/tests_launch_servo_cpp_integration.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_servo/test_results/moveit_servo/tests_launch_servo_utils.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/moveit_setup_app_plugins/Testing/20240528-1751/Test.xml: 1 test, 0 errors, 1 failure, 0 skipped
build/moveit_setup_controllers/Testing/20240528-1751/Test.xml: 1 test, 0 errors, 1 failure, 0 skipped
build/moveit_setup_srdf_plugins/Testing/20240528-1751/Test.xml: 1 test, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/Testing/20240528-1751/Test.xml: 17 tests, 0 errors, 10 failures, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_joint_limits_aggregator.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_pilz_industrial_motion_planner.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_planning_context.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_planning_context_loaders.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_blender_transition_window.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_functions.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_generator_circ.test.py.xunit.xml: 2 tests, 0 errors, 1 failure, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_generator_common.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_generator_lin.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
build/pilz_industrial_motion_planner/test_results/pilz_industrial_motion_planner/test_unit_tests_launch_unittest_trajectory_generator_ptp.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
Summary: 679 tests, 5 errors, 37 failures, 59 skippedThat's all at the moment in testing side. I need to work more on this errors to give more information about that subject.
CihatAltiparmak
commented
4 months ago For benchmarking, i just have created this repository in my account and i will develop benchmark code on this repository until the end of GSOC project. Here is our repository. I give a link for those who wanted to track the zenoh benchmarking progress.
https://github.com/CihatAltiparmak/moveit_middleware_benchmark
I need to refactor some part of repository at the moment, however i will. We want that our benchmarks is based on rmw_zenoh. Whereas we can switch to dds-based middleware + zenoh_dds_bridge combination rather than rmw_zenoh due to the fact that rmw_zenoh is the experimental middleware and still has some bugs. This doesn't mean that our benchmark is not based on rmw_zenoh. We will continue to track the development progress of rmw_zenoh closely. Maybe we can fail in this progress in side of benchmark of rmw_zenoh but i will try to implement this benchmark codes in a way it can be able to runnable even after GSOC project.
CihatAltiparmak
commented
3 months ago Hello all,
We want to share some updates related to this project.
When we ran MoveIt with rmw_zenoh middleware, we observed some stuck in the action clients and this stuck occurs due to rmw_wait having a data race. This seems related to rmw_zenoh, not MoveIt. Additionaly, the reason of this stuck seems MultiThreadedExecutor. rmw_zenoh seems to work without problem with SingleThreadedExecutor but has problems with MultiThreadedExecutor. For those seeking more information about this problem, here is some additional content. It should also be mentioned that it seems navigation2 works well with rmw_zenoh especially in this webinar. when I searched whether navigation2 uses MultiThreadedExecutor, i saw that navigation2 doesn't use MultiThreadedExecutor but uses SingleThreadedExecutor while MoveIt uses MultiThreadedExecutor in some part of code. Furthermore, this stucks helped us to discover some bad implementations inside MoveIt. I'm still working on a better implementation. We also discussed trying rmw_zenoh in MoveIt, modified in a way to use another private node spinning technique instead of MultiThreadedExecutor, both to avoid breaking the original MoveIt flow and to enable the use of rmw_zenoh without problem. By the way there are two upstream in rmw_zenoh to fix this bug. I tried it but yet to run with MoveIt. You can find these upstreams here.
We are also working on benchmarking the effects of middleware in MoveIt. We implemented the benchmark code for perception pipeline in this development branch. We will add more scenario to this repository.
CihatAltiparmak
commented
3 months ago Hello all,
In today's meeting, one of the topics we talked about was rmw_zenoh integration to MoveIt. I was able to successfully run MoveIt using rmw zenoh with some modifications. You can visit this draft pull request to see what's changed. However i must say that this is experimental pull request and it can be observed some unexpected effects. In order to run MoveIt using rmw_zenoh, you can use my fork at the moment. I am still waiting for better rmw_wait implementation and following rmw_zenoh progress. I also created an issue related to TrajectoryExecutionManager.
For benchmarking, unfortunately we have some problems. I don't want to share middleware benchmarking results for perception pipeline scenario at the moment and we are investigating it. CPU time is much less than Real Time in our benchmarking results. That's problem. On the other side, we also discussed about benchmarking memory usage and I suggested this repository.
CihatAltiparmak
commented
1 month ago Hello everyone,
I want to share some updates about benchmarking side. The middleware benchmarking tool is still being developed rollingly by us. I conducted some benchmarks for two scenarios, one of them is a basic service-client benchmark and other one is perception pipeline (we couldn't conduct this benchmark using camera topic but it can be conducted using bag file.). You can take a look at some documentation here to get what corresponding scenario aims. Seems like rmw_zenoh (https://github.com/ros2/rmw_zenoh/commit/c12ff3ed2b73169ff0bd7a2a341f8911082fc1e9) is as fast as rmw_fastrtps with some configurations in tcp. I strongly suggest you to visit this thread in Zenoh's discord community.
sudo sysctl -w "net.ipv4.tcp_rmem=4096 4096 4096"
sudo sysctl -w "net.ipv4.tcp_wmem=4096 4096 4096"
sudo sysctl -w "net.ipv4.tcp_mem=4096 4096 4096"This is the initial benchmark results:
I couldn't understand why rmw_zenoh is dramatically slower than rmw_fastrtps. I will investigate this problem. But i should also say that rmw_zenoh defaultly uses TCP whereas rmw_cyclonedds uses UDP. I will conduct this benchmarks by configuring rmw_zenoh to UDP and let you to know it in second post.
I have also worked on visualization side and CI side to conduct benchmarks in github CI. This website doesn't present the real benchmarking results. It's just for dummy visualization. When we merge other PR's in moveit_middleware_benchmark repository, we will be able to start to get some visualizations. Here is the example website. https://darkusalphahydranoid.github.io/moveit_middleware_benchmark_experimental
By the way you can try our benchmark tool by pulling its docker image like below.
Terminal 1
docker pull ghcr.io/cihataltiparmak/moveit_middleware_benchmark:latest
# run docker image
# inside docker
cd /ws_moveit
source /opt/ros/rolling/setup.bash
source install/setup.bash
export RMW_IMPLEMENTATION=rmw_zenoh_cpp
ros2 launch moveit_middleware_benchmark scenario_basic_service_client_benchmark.launch.pyTerminal 2
docker exec -it <container_id> bash
# inside docker
cd /ws_moveit
source /opt/ros/rolling/setup.bash
source install/setup.bash
export RMW_IMPLEMENTATION=rmw_zenoh_cpp
ros2 run rmw_zenoh_cpp rmw_zenohdDon't be hesitate to ask me about anything related to this GSOC project (benchmarking, bugs in rmw_zenoh and so on). I'm ready to answer you. This is my mail : cihataltiparmak1@gmail.com
Hello everyone,
I will work on integrating rmw_zenoh middleware to MoveIt and benchmarking the effects of Zenoh at MoveIt in scope of the Google Summer of Code Project with my mentors @sjahr and @henningkayser . We will share this project's updates here.
Our goals are below.
The primary goals of these project are above. However, please don't hesitate to share your ideas. I am open to new ideas.