search

ivaROS / mbplanner_ros

Motion-primitives Based Planner for Fast & Agile Exploration
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Segfault #1

Open drjsmith opened 2 years ago

drjsmith commented 2 years ago

Every time I run the planner, it eventually segfaults:

[ INFO] [1662672741.116560770, 694.010000000]: Reseting the tree.
[ INFO] [1662672741.116667292, 694.010000000]: Tree root:
[ INFO] [1662672741.116727517, 694.010000000]: 
       Position      :  x: 186.751372  y: -0.815942  z: 0.903084 
       Velocity      : vx: 1.727866 vy: -0.178220 vz: 0.212643 
       Acceleration  : ax: 0.000000 ay: 0.000000 az: 0.000000
[ WARN] [1662672741.118668200, 694.010000000]: ROOT INSIDE OBSTACLE
--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 "mbplanner_node" received signal SIGSEGV, Segmentation fault.
std::__uninitialized_copy_a<__gnu_cxx::__normal_iterator<Eigen::Matrix<int, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > >, Eigen::Matrix<int, 3, 1, 0, 3, 1>*, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > (__alloc=..., __result=0x0, __last=..., __first=...)
    at /usr/include/c++/9/bits/stl_uninitialized.h:293
293                 __traits::construct(__alloc, std::__addressof(*__cur), *__first);
(gdb) bt
#0  std::__uninitialized_copy_a<__gnu_cxx::__normal_iterator<Eigen::Matrix<int, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > >, Eigen::Matrix<int, 3, 1, 0, 3, 1>*, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > (__alloc=..., __result=0x0, __last=..., __first=...)
    at /usr/include/c++/9/bits/stl_uninitialized.h:293
#1  std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > >::vector (__x=<error reading variable: Cannot access memory at address 0x20>, 
    this=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:555
#2  std::pair<Eigen::Matrix<int, 3, 1, 0, 3, 1>, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > >::pair<Eigen::Matrix<int, 3, 1, 0, 3, 1> const, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > >, true> (__p={...}, this=<optimized out>)
    at /usr/include/c++/9/bits/stl_pair.h:292

#3  explorer::MapManagerVoxblox<voxblox::TsdfServer, voxblox::TsdfVoxel>::augmentFreeBox (this=0x555555ae19b0, position=..., box_size=...)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/planner_common/src/map_manager_voxblox_impl.cpp:277
#4  0x00007ffff7ca7fbe in explorer::mbplanner::MBTree::buildTree (this=0x555555adf8e0, prev_point_plan=prev_point_plan@entry=false, use_current_state=use_current_state@entry=false, 
    use_given_vel=use_given_vel@entry=false) at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbtree.cpp:221
#5  0x00007ffff7c929b4 in explorer::MBPlanner::plannerServiceCallback (this=0x7fffffff8690, req=..., res=...)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbplanner.cpp:73
#6  0x00007ffff7c9f34b in boost::function2<bool, planner_msgs::planner_srvRequest_<std::allocator<void> >&, planner_msgs::planner_srvResponse_<std::allocator<void> >&>::operator() (a1=..., 
    a0=..., this=0x555555b1ab68) at /usr/include/boost/function/function_template.hpp:677
#7  ros::ServiceSpec<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > >::call(boost::function<bool (planner_msgs::planner_srvRequest_<std::allocator<void> >&, planner_msgs::planner_srvResponse_<std::allocator<void> >&)> const&, ros::ServiceSpecCallParams<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > >&) (params=<synthetic pointer>..., cb=...) at /home/justin/workspaces/ros_catkin_ws/rinstall/include/ros/service_callback_helper.h:125
#8  ros::ServiceCallbackHelperT<ros::ServiceSpec<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > > >::call (
    this=0x555555b1ab60, params=...) at /home/justin/workspaces/ros_catkin_ws/rinstall/include/ros/service_callback_helper.h:182
#9  0x00007ffff7d77b55 in ros::ServiceCallback::call (this=0x7fffe0029520) at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/service_publication.cpp:125
#10 0x00007ffff7dc9e82 in ros::CallbackQueue::callOneCB (this=0x555555ad5410, tls=0x555555b1b7e0)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/callback_queue.cpp:410
#11 0x00007ffff7dcb4b3 in ros::CallbackQueue::callAvailable (this=this@entry=0x555555ad5410, timeout=...)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/callback_queue.cpp:351
#12 0x00007ffff7e1d2ff in ros::SingleThreadedSpinner::spin (this=<optimized out>, queue=<optimized out>)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/spinner.cpp:141
#13 0x00007ffff7e05d4f in ros::spin () at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/init.cpp:563
#14 0x000055555555dca7 in main (argc=<optimized out>, argv=<optimized out>)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbplanner_ros_node.cpp:143
(gdb) 
#0  std::__uninitialized_copy_a<__gnu_cxx::__normal_iterator<Eigen::Matrix<int, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > >, Eigen::Matrix<int, 3, 1, 0, 3, 1>*, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > (__alloc=..., __result=0x0, __last=..., __first=...)
    at /usr/include/c++/9/bits/stl_uninitialized.h:293
#1  std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > >::vector (__x=<error reading variable: Cannot access memory at address 0x20>, 
    this=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:555
#2  std::pair<Eigen::Matrix<int, 3, 1, 0, 3, 1>, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > > >::pair<Eigen::Matrix<int, 3, 1, 0, 3, 1> const, std::vector<Eigen::Matrix<int, 3, 1, 0, 3, 1>, Eigen::aligned_allocator<Eigen::Matrix<int, 3, 1, 0, 3, 1> > >, true> (__p={...}, this=<optimized out>)
    at /usr/include/c++/9/bits/stl_pair.h:292
#3  explorer::MapManagerVoxblox<voxblox::TsdfServer, voxblox::TsdfVoxel>::augmentFreeBox (this=0x555555ae19b0, position=..., box_size=...)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/planner_common/src/map_manager_voxblox_impl.cpp:277
#4  0x00007ffff7ca7fbe in explorer::mbplanner::MBTree::buildTree (this=0x555555adf8e0, prev_point_plan=prev_point_plan@entry=false, use_current_state=use_current_state@entry=false, 
    use_given_vel=use_given_vel@entry=false) at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbtree.cpp:221
#5  0x00007ffff7c929b4 in explorer::MBPlanner::plannerServiceCallback (this=0x7fffffff8690, req=..., res=...)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbplanner.cpp:73
#6  0x00007ffff7c9f34b in boost::function2<bool, planner_msgs::planner_srvRequest_<std::allocator<void> >&, planner_msgs::planner_srvResponse_<std::allocator<void> >&>::operator() (a1=..., 
    a0=..., this=0x555555b1ab68) at /usr/include/boost/function/function_template.hpp:677
#7  ros::ServiceSpec<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > >::call(boost::function<bool (planner_msgs::planner_srvRequest_<std::allocator<void> >&, planner_msgs::planner_srvResponse_<std::allocator<void> >&)> const&, ros::ServiceSpecCallParams<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > >&) (params=<synthetic pointer>..., cb=...) at /home/justin/workspaces/ros_catkin_ws/rinstall/include/ros/service_callback_helper.h:125
#8  ros::ServiceCallbackHelperT<ros::ServiceSpec<planner_msgs::planner_srvRequest_<std::allocator<void> >, planner_msgs::planner_srvResponse_<std::allocator<void> > > >::call (
    this=0x555555b1ab60, params=...) at /home/justin/workspaces/ros_catkin_ws/rinstall/include/ros/service_callback_helper.h:182
#9  0x00007ffff7d77b55 in ros::ServiceCallback::call (this=0x7fffe0029520) at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/service_publication.cpp:125
#10 0x00007ffff7dc9e82 in ros::CallbackQueue::callOneCB (this=0x555555ad5410, tls=0x555555b1b7e0)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/callback_queue.cpp:410
#11 0x00007ffff7dcb4b3 in ros::CallbackQueue::callAvailable (this=this@entry=0x555555ad5410, timeout=...)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/callback_queue.cpp:351
#12 0x00007ffff7e1d2ff in ros::SingleThreadedSpinner::spin (this=<optimized out>, queue=<optimized out>)
    at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/spinner.cpp:141
#13 0x00007ffff7e05d4f in ros::spin () at /home/justin/workspaces/ros_catkin_ws/src/ros_comm/roscpp/src/libros/init.cpp:563
#14 0x000055555555dca7 in main (argc=<optimized out>, argv=<optimized out>)
    at /home/justin/workspaces/mbplanner_catin_ws/src/src/exploration/mbplanner_ros/mbplanner/src/mbplanner_ros_node.cpp:143
(gdb)

It used to crash due to a memory error (double free detected tcache 2); I recompiled the whole workspace with -DEIGEN_DONT_VECTORIZE, and haven't seen that error since.

IIRC, the classes aren't defining EIGEN_MAKE_ALIGNED_OPERATOR_NEW, so maybe that's the problem? Though I would have thought that not vectorizing would solve it too.

drjsmith commented 1 year ago

A recent crash has been potentially more informative.

The planner crashed when the robot was in this state: image image

A successful planning cycle looks something like this:

[ WARN] [1674238243.761970926, 821.310000000]: [GlobalGraph] Try to add current state to the graph.
[ WARN] [1674238243.765613563, 821.310000000]: [GlobalGraph] Added successfully.
[ INFO] [1674238243.785546063, 821.320000000]: Global graph: 1817 vertices, 7511 edges.
[ INFO] [1674238243.785883710, 821.320000000]: Have 226 frontiers from global graph.
[ INFO] [1674238255.735007191, 831.210000000]: Waiting done.
[ WARN] [1674238255.735108304, 831.210000000]: PCI: Ready to trigger the planner.
[ INFO] [1674238255.735185888, 831.210000000]: PlannerControlInterface: Running Planner (kAuto)
[ INFO] [1674238255.763959286, 831.220000000]: Planning iteration 34
[ INFO] [1674238255.765199049, 831.220000000]: Reseting the tree.
[ INFO] [1674238255.765279195, 831.220000000]: Tree root:
[ INFO] [1674238255.765331584, 831.220000000]: 
       Position      :  x: 214.254343  y: 50.393582  z: 1.560330 
       Velocity      : vx: -0.478844 vy: 1.681751 vz: -0.070147 
       Acceleration  : ax: 0.000000 ay: 0.000000 az: 0.000000
[ INFO] [1674238256.055944732, 831.470000000]: Tree built with [286] nodes
[ INFO] [1674238256.472811074, 831.820000000]: Number of leaves: 149
[ INFO] [1674238256.552980072, 831.900000000]: Valid paths: 11, invalid paths: 0
[ INFO] [1674238256.553066211, 831.900000000]: Similar paths published
[ INFO] [1674238256.553334992, 831.900000000]: Tree eval visualized
[ INFO] [1674238256.553768092, 831.900000000]: Time statistics:
         Build tree     : 0.250 (s)
         Compute gain   : 0.350 (s)
         Evaluate graph : 0.080 (s)
         Total          : 0.680 (s)
[ INFO] [1674238256.553831935, 831.900000000]: Time Elapsed: 831.900000
[ INFO] [1674238256.732388133, 832.040000000]: LOCAL PLANNER SUCCESSFULLY FOUND PATH
[ INFO] [1674238256.740265826, 832.050000000]: Reconnect done

Contrast that with the fatal cycle:

[ WARN] [1674238270.651361470, 843.660000000]: [GlobalGraph] Try to add current state to the graph.
[ WARN] [1674238270.654770183, 843.670000000]: [GlobalGraph] Added successfully.
[ INFO] [1674238270.671737051, 843.690000000]: Global graph: 1941 vertices, 7951 edges.
[ INFO] [1674238270.672234055, 843.690000000]: Have 247 frontiers from global graph.
[ INFO] [1674238282.693762711, 853.960000000]: Waiting done.
[ WARN] [1674238282.693881194, 853.960000000]: PCI: Ready to trigger the planner.
[ INFO] [1674238282.693969470, 853.960000000]: PlannerControlInterface: Running Planner (kAuto)
[ INFO] [1674238282.703634265, 853.970000000]: Planning iteration 36
[ INFO] [1674238283.067007617, 854.270000000]: Reseting the tree.
[ INFO] [1674238283.067092881, 854.270000000]: Tree root:
[ INFO] [1674238283.067140881, 854.270000000]: 
       Position      :  x: 214.431501  y: 70.537099  z: 0.748890 
       Velocity      : vx: -0.394150 vy: 1.697412 vz: -0.161053 
       Acceleration  : ax: 0.000000 ay: 0.000000 az: 0.000000
[ WARN] [1674238283.067627880, 854.270000000]: ROOT INSIDE OBSTACLE
[ WARN] [1674238283.465492901, 854.630000000]: Planner service failed
[mbplanner_node-14] process has died [pid 2894455, exit code -11,

The coordinates of the new tree root appear to match up with those of the final node on the exploration graph (the upper-left-most of the larger blue spheres, a little to the left of the cyan arrow). image

Click here for a larger portion of the logs ``` Velocity : vx: -0.236266 vy: 1.733739 vz: 0.028758 Acceleration : ax: 0.000000 ay: 0.000000 az: 0.000000 [ INFO] [1674238242.723648229, 820.440000000]: Tree built with [318] nodes [ INFO] [1674238243.212610355, 820.860000000]: Number of leaves: 162 [ INFO] [1674238243.305894287, 820.940000000]: Valid paths: 25, invalid paths: 0 [ INFO] [1674238243.305973845, 820.940000000]: Similar paths published [ INFO] [1674238243.306288174, 820.940000000]: Tree eval visualized [ INFO] [1674238243.306715201, 820.940000000]: Time statistics: Build tree : 0.290 (s) Compute gain : 0.420 (s) Evaluate graph : 0.080 (s) Total : 0.790 (s) [ INFO] [1674238243.306774655, 820.940000000]: Time Elapsed: 820.940000 [ INFO] [1674238243.486320081, 821.100000000]: LOCAL PLANNER SUCCESSFULLY FOUND PATH [ INFO] [1674238243.495374305, 821.110000000]: Reconnect done [ WARN] [1674238243.761970926, 821.310000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238243.765613563, 821.310000000]: [GlobalGraph] Added successfully. [ INFO] [1674238243.785546063, 821.320000000]: Global graph: 1817 vertices, 7511 edges. [ INFO] [1674238243.785883710, 821.320000000]: Have 226 frontiers from global graph. [ INFO] [1674238255.735007191, 831.210000000]: Waiting done. [ WARN] [1674238255.735108304, 831.210000000]: PCI: Ready to trigger the planner. [ INFO] [1674238255.735185888, 831.210000000]: PlannerControlInterface: Running Planner (kAuto) [ INFO] [1674238255.763959286, 831.220000000]: Planning iteration 34 [ INFO] [1674238255.765199049, 831.220000000]: Reseting the tree. [ INFO] [1674238255.765279195, 831.220000000]: Tree root: [ INFO] [1674238255.765331584, 831.220000000]: Position : x: 214.254343 y: 50.393582 z: 1.560330 Velocity : vx: -0.478844 vy: 1.681751 vz: -0.070147 Acceleration : ax: 0.000000 ay: 0.000000 az: 0.000000 [ INFO] [1674238256.055944732, 831.470000000]: Tree built with [286] nodes [ INFO] [1674238256.472811074, 831.820000000]: Number of leaves: 149 [ INFO] [1674238256.552980072, 831.900000000]: Valid paths: 11, invalid paths: 0 [ INFO] [1674238256.553066211, 831.900000000]: Similar paths published [ INFO] [1674238256.553334992, 831.900000000]: Tree eval visualized [ INFO] [1674238256.553768092, 831.900000000]: Time statistics: Build tree : 0.250 (s) Compute gain : 0.350 (s) Evaluate graph : 0.080 (s) Total : 0.680 (s) [ INFO] [1674238256.553831935, 831.900000000]: Time Elapsed: 831.900000 [ INFO] [1674238256.732388133, 832.040000000]: LOCAL PLANNER SUCCESSFULLY FOUND PATH [ INFO] [1674238256.740265826, 832.050000000]: Reconnect done [ WARN] [1674238257.620857953, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.620975041, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.621102662, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.621162908, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.621233775, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.621288397, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.621347440, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.621402419, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.621563575, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.621650176, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.621871510, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.621942784, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ WARN] [1674238257.622170718, 832.790000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238257.622229760, 832.790000000]: [GlobalGraph] Can not add current state to the global graph. [ INFO] [1674238257.622263175, 832.790000000]: Global graph: 1887 vertices, 7748 edges. [ INFO] [1674238257.622448958, 832.790000000]: Have 245 frontiers from global graph. [ INFO] [1674238269.185849914, 842.450000000]: Waiting done. [ WARN] [1674238269.185981524, 842.450000000]: PCI: Ready to trigger the planner. [ INFO] [1674238269.186063259, 842.450000000]: PlannerControlInterface: Running Planner (kAuto) [ INFO] [1674238269.195730947, 842.460000000]: Planning iteration 35 [ INFO] [1674238269.196081832, 842.460000000]: Reseting the tree. [ INFO] [1674238269.196189707, 842.460000000]: Tree root: [ INFO] [1674238269.196282424, 842.460000000]: Position : x: 212.719690 y: 60.429623 z: 0.927535 Velocity : vx: 0.287118 vy: 1.724446 vz: 0.079694 Acceleration : ax: 0.000000 ay: 0.000000 az: 0.000000 [ INFO] [1674238269.628901557, 842.810000000]: Tree built with [218] nodes [ INFO] [1674238269.994324143, 843.110000000]: Number of leaves: 126 [ INFO] [1674238270.068067388, 843.190000000]: Valid paths: 1, invalid paths: 0 [ INFO] [1674238270.068139655, 843.190000000]: Similar paths published [ INFO] [1674238270.068358742, 843.190000000]: Tree eval visualized [ INFO] [1674238270.068769132, 843.190000000]: Time statistics: Build tree : 0.350 (s) Compute gain : 0.300 (s) Evaluate graph : 0.080 (s) Total : 0.730 (s) [ INFO] [1674238270.068806559, 843.190000000]: Time Elapsed: 843.190000 [ INFO] [1674238270.272745165, 843.350000000]: LOCAL PLANNER SUCCESSFULLY FOUND PATH [ INFO] [1674238270.281654555, 843.360000000]: Reconnect done [ WARN] [1674238270.651361470, 843.660000000]: [GlobalGraph] Try to add current state to the graph. [ WARN] [1674238270.654770183, 843.670000000]: [GlobalGraph] Added successfully. [ INFO] [1674238270.671737051, 843.690000000]: Global graph: 1941 vertices, 7951 edges. [ INFO] [1674238270.672234055, 843.690000000]: Have 247 frontiers from global graph. [ INFO] [1674238282.693762711, 853.960000000]: Waiting done. [ WARN] [1674238282.693881194, 853.960000000]: PCI: Ready to trigger the planner. [ INFO] [1674238282.693969470, 853.960000000]: PlannerControlInterface: Running Planner (kAuto) [ INFO] [1674238282.703634265, 853.970000000]: Planning iteration 36 [ INFO] [1674238283.067007617, 854.270000000]: Reseting the tree. [ INFO] [1674238283.067092881, 854.270000000]: Tree root: [ INFO] [1674238283.067140881, 854.270000000]: Position : x: 214.431501 y: 70.537099 z: 0.748890 Velocity : vx: -0.394150 vy: 1.697412 vz: -0.161053 Acceleration : ax: 0.000000 ay: 0.000000 az: 0.000000 [ WARN] [1674238283.067627880, 854.270000000]: ROOT INSIDE OBSTACLE [ WARN] [1674238283.465492901, 854.630000000]: Planner service failed [mbplanner_node-14] process has died [pid 2894455, exit code -11, cmd /home/justin/workspaces/mbplanner_catin_ws/rdevel/lib/mbplanner/mbplanner_node odometry:=m100/ground_truth/odometry_throttled pointcloud:=/velodyne_points __name:=mbplanner_node __log:=/home/justin/.ros/log/afea3fec-98e3-11ed-bf6c-5d865e3fd7b9/mbplanner_node-14.log]. log file: /home/justin/.ros/log/afea3fec-98e3-11ed-bf6c-5d865e3fd7b9/mbplanner_node-14*.log [ INFO] [1674238284.031354125, 855.140000000]: Planning iteration 36 [ INFO] [1674238284.623442951, 855.650000000]: Planning iteration 36 [ WARN] [1674238284.623536727, 855.650000000]: Planner service failed [ INFO] [1674238285.196201137, 856.150000000]: PlannerControlInterface: Running Planner (kAuto) [ INFO] [1674238285.204823269, 856.160000000]: Planning iteration 36 [ INFO] [1674238285.804563053, 856.670000000]: Planning iteration 36 [ WARN] [1674238285.804652928, 856.670000000]: Planner service failed [ INFO] [1674238286.381975041, 857.180000000]: Planning iteration 36 [ INFO] [1674238286.941685928, 857.680000000]: PlannerControlInterface: Running Planner (kAuto) ```

This test was preceded by several that also failed, though much earlier. I had observed at the time that the robot appeared to have collided with the wall or maybe floor, but looking back through the logs of the 2 previous failures, I found that they too crashed right after the ROOT INSIDE OBSTACLE warnings.

First, I will look into how the ROOT INSIDE OBSTACLE condition is handled. Afterwards, I can investigate why colliding nodes are added to the tree in the first place.