search

moveit / moveit_task_constructor

A hierarchical multi-stage manipulation planner
https://moveit.github.io/moveit_task_constructor
BSD 3-Clause "New" or "Revised" License
176 stars 150 forks source link

Rviz task constructor running out of available threads #404

Closed DatSpace closed 1 year ago

DatSpace commented 1 year ago

I am planning with task constructor again and again. On RViz the task are added normally, but because I might be planning 1 task per second, it adds up pretty quickly.

It seems that for every task published a thread is created to hold all the trajectories etc which is understandable but it quickly builds up the thread count. On my setup rviz crashes with:

[rviz2-5] 1670093332.162819 [] rviz2: create_thread: rviz2: no free slot

at around 105 threads on the process. The memory used by the rviz process is more than 2GB, while at the start its less than 1GB.

While the plans are added to the panel, if I select all and clear the threads are destroyed and the memory drops, but manually clearing is not an option.

1) Is that a known bug and is there something I can do to avoid it ? 2) Is there a function I can call to clear the task from the panel programmatically? 3) How can I disable the publishing of the solutions at all for a task? a) I tried disabling introspection but I also need to be able to access the stage id's which I believe are not available: moveit_task_constructor_msgs::msg::Solution solution; task->solutions().front()->fillMessage(solution, &task->introspection());

Using ros2 humble, with the latest released moveit and latest task constructor in a doker container.

rhaschke commented 1 year ago
  • Is that a known bug and is there something I can do to avoid it ?

I was not aware that new incoming tasks create new threads. Could you send a backtrace of this, please, to understand where the thread gets created? The error msg is related to CycloneDDS: https://github.com/ros2/rmw_cyclonedds/issues/268

  • Is there a function I can call to clear the task from the panel programmatically?

In the TaskPanel settings you can configure how to deal with old tasks. You can automatically remove them, for example.

  • How can I disable the publishing of the solutions at all for a task?

Simply disable introspection: task->enableIntrospection(false);

DatSpace commented 1 year ago

Thank you for your answer. Here is the rviz backtrace:

Backtrace ``` [New Thread 0x7f17997fa640 (LWP 1552188)] [WARN] [1670280726.193011661] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140632258489920 [New Thread 0x7f1798ff9640 (LWP 1552528)] [New Thread 0x7f177bfff640 (LWP 1552831)] [New Thread 0x7f177b7fe640 (LWP 1553112)] [New Thread 0x7f177affd640 (LWP 1553555)] [New Thread 0x7f177a7fc640 (LWP 1553809)] [WARN] [1670280728.781763919] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140632122842784 [WARN] [1670280728.895899992] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140632122842784 [New Thread 0x7f1779ffb640 (LWP 1554197)] [New Thread 0x7f17797fa640 (LWP 1554454)] [New Thread 0x7f1778ff9640 (LWP 1554971)] [INFO] [1670280730.659388698] [rviz2]: Message Filter dropping message: frame '' at time 0.000 for reason 'the frame id of the message is empty' [New Thread 0x7f175bfff640 (LWP 1555338)] [New Thread 0x7f175b7fe640 (LWP 1555677)] [New Thread 0x7f175affd640 (LWP 1556063)] [New Thread 0x7f175a7fc640 (LWP 1556166)] [New Thread 0x7f1759ffb640 (LWP 1556449)] [New Thread 0x7f17597fa640 (LWP 1556831)] [New Thread 0x7f1758ff9640 (LWP 1557253)] [New Thread 0x7f173bfff640 (LWP 1557678)] [New Thread 0x7f173b7fe640 (LWP 1558079)] [New Thread 0x7f173affd640 (LWP 1558382)] [New Thread 0x7f173a7fc640 (LWP 1558776)] [New Thread 0x7f1739ffb640 (LWP 1559209)] [New Thread 0x7f17397fa640 (LWP 1559563)] [WARN] [1670280737.975169593] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140631855244720 [WARN] [1670280738.153771093] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140631855244720 [New Thread 0x7f1738ff9640 (LWP 1560003)] [New Thread 0x7f171bfff640 (LWP 1560327)] [New Thread 0x7f171b7fe640 (LWP 1560643)] [New Thread 0x7f171affd640 (LWP 1561087)] [New Thread 0x7f171a7fc640 (LWP 1561528)] [WARN] [1670280740.784841234] [moveit_task_constructor_visualization.task_list_model]: unknown task: 3f771b6d32ae_1482624_140632589754720 [New Thread 0x7f1719ffb640 (LWP 1561804)] [New Thread 0x7f17197fa640 (LWP 1562079)] 1670280741.459901 [] rviz2: create_thread: rviz2: no free slot Thread 134 "rviz2" received signal SIGABRT, Aborted. [Switching to Thread 0x7f17197fa640 (LWP 1562079)] __pthread_kill_implementation (no_tid=0, signo=6, threadid=139737188771392) at ./nptl/pthread_kill.c:44 44 ./nptl/pthread_kill.c: No such file or directory. (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=139737188771392) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=139737188771392) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=139737188771392, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007f198473f476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007f19847257f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x00007f197ef6724a in vlog (ap=0x7f17197f6780, fmt=0x7f197f02ca58 "create_thread: %s: no free slot\n", func=0x7f197f02da00 <__func__.5.lto_priv.7> "init_thread_state", line=266, file=0x7f197f02c9b0 "./src/core/ddsi/src/q_thread.c", domid=4294967295, cat=, cfg=) at ./src/ddsrt/src/log.c:302 #6 dds_log (cat=, file=0x7f197f02c9b0 "./src/core/ddsi/src/q_thread.c", line=266, func=0x7f197f02da00 <__func__.5.lto_priv.7> "init_thread_state", fmt=0x7f197f02ca58 "create_thread: %s: no free slot\n") at ./src/ddsrt/src/log.c:334 #7 0x00007f197efcc4a7 in init_thread_state ( tname=tname@entry=0x7f17197f68b0 "rviz2", gv=gv@entry=0x0, state=state@entry=THREAD_STATE_LAZILY_CREATED) at ./src/core/ddsi/src/q_thread.c:266 --Type for more, q to quit, c to continue without paging--c #8 0x00007f197efcc58e in lazy_create_thread_state (self=...) at ./src/core/ddsi/src/q_thread.c:205 #9 lookup_thread_state_real () at ./src/core/ddsi/src/q_thread.c:223 #10 0x00007f197efeb0f5 in lookup_thread_state () at ./src/security/api/../../core/ddsi/include/dds/ddsi/q_thread.h:139 #11 dds_read_impl (take=take@entry=true, reader_or_condition=379082681, buf=0x7f17197f6a18, bufsz=1, maxs=1, si=0x7f17197f6a20, mask=128, hand=0, lock=true, only_reader=false) at ./src/core/ddsc/src/dds_read.c:36 #12 0x00007f197efeb59b in dds_take (rd_or_cnd=, buf=, si=, bufsz=, maxs=) at ./src/core/ddsc/src/dds_read.c:369 #13 0x00007f197f098a65 in rmw_take_int (subscription=0x5598773e9e70, ros_message=, taken=0x7f17197f6b1f, message_info=0x7f17197f88e8) at ./src/rmw_node.cpp:3154 #14 0x00007f19837ed3ba in rcl_take () from /opt/ros/humble/lib/librcl.so #15 0x00007f1984ce705f in rclcpp::SubscriptionBase::take_type_erased(void*, rclcpp::MessageInfo&) () from /opt/ros/humble/lib/librclcpp.so #16 0x00007f1984c6e301 in ?? () from /opt/ros/humble/lib/librclcpp.so #17 0x00007f1984c6e87c in rclcpp::Executor::execute_subscription(std::shared_ptr) () from /opt/ros/humble/lib/librclcpp.so #18 0x00007f1984c6f07f in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) () from /opt/ros/humble/lib/librclcpp.so #19 0x00007f1984c709c1 in rclcpp::Executor::spin_once_impl(std::chrono::duration >) () from /opt/ros/humble/lib/librclcpp.so #20 0x00007f1984d02e61 in ?? () from /opt/ros/humble/lib/librclcpp.so #21 0x00007f1984a232b3 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #22 0x00007f1984791b43 in start_thread (arg=) at ./nptl/pthread_create.c:442 #23 0x00007f1984822bb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 ```
rhaschke commented 1 year ago

Looks like CycloneDDS can only handle a fixed amount of threads. For each incoming message a new thread is started to process that message. Closing here as this isn‘t a bug in MTC.