`behavior_planning_container` dies while driving straight.

Kim-mins commented 2 months ago

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I'm convinced that this is not my fault but a bug.

Description

Hi team,

I'm currently running Autoware with Carla using bridge implementation here, and I found a situation that the behavior_planning_container diles while driving straight, and it makes the ego vehicle stop before reaching the goal.

Here's the video of the situation: [rviz] In the video, the vehicle drive well, but it suddenly stops at 0:30.

I also checked a corresponding [launch.log] file and I could find the error as below:

...
1721395398.4731574 [component_container_mt-75] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
1721395398.4732425 [component_container_mt-75]   what():  failed to add guard condition to wait set: guard condition implementation is invalid, at ./src/rcl/guard_condition.c:172, at ./src/rcl/wait.c:460
1721395398.4733043 [component_container_mt-75] *** Aborted at 1721395398 (unix time) try "date -d @1721395398" if you are using GNU date ***
1721395398.4819667 [component_container_mt-75] PC: @                0x0 (unknown)
1721395398.4821553 [component_container_mt-75] *** SIGABRT (@0x3e80000136d) received by PID 4973 (TID 0x7fc3933da640) from PID 4973; stack trace: ***
1721395398.4822626 [component_container_mt-75]     @     0x7fc304040a86 google::(anonymous namespace)::FailureSignalHandler()
1721395398.4823201 [component_container_mt-75]     @     0x7fc39d27b520 (unknown)
1721395398.4841051 [component_container_mt-75]     @     0x7fc39d2cf9fc pthread_kill
1721395398.4870968 [component_container_mt-75]     @     0x7fc39d27b476 raise
1721395398.4893513 [component_container_mt-75]     @     0x7fc39d2617f3 abort
1721395398.4922843 [component_container_mt-75]     @     0x7fc39d524b9e (unknown)
1721395398.4951425 [component_container_mt-75]     @     0x7fc39d53020c (unknown)
1721395398.4973755 [component_container_mt-75]     @     0x7fc39d530277 std::terminate()
1721395398.4996758 [component_container_mt-75]     @     0x7fc39d5301fe std::rethrow_exception()
1721395398.5018551 [component_container_mt-75]     @     0x7fc39d7cf119 rclcpp::exceptions::throw_from_rcl_error()
1721395398.5039811 [component_container_mt-75]     @     0x7fc39d7d2e9c rclcpp::detail::add_guard_condition_to_rcl_wait_set()
1721395398.5060802 [component_container_mt-75]     @     0x7fc39d7e55d7 (unknown)
1721395398.5080755 [component_container_mt-75]     @     0x7fc39d7d2685 rclcpp::Executor::wait_for_work()
1721395398.5100472 [component_container_mt-75]     @     0x7fc39d7d53c3 rclcpp::Executor::get_next_executable()
1721395398.5119836 [component_container_mt-75]     @     0x7fc39d7dc252 rclcpp::executors::MultiThreadedExecutor::run()
1721395398.5138829 [component_container_mt-75]     @     0x7fc39d55e253 (unknown)
1721395398.5156958 [component_container_mt-75]     @     0x7fc39d2cdac3 (unknown)
1721395398.5174739 [component_container_mt-75]     @     0x7fc39d35f850 (unknown)
1721395398.5193157 [component_container_mt-75]     @                0x0 (unknown)
1721395398.6333418 [ERROR] [component_container_mt-75]: process has died [pid 4973, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=behavior_planning_container -r __ns:=/planning/scenario_planning/lane_driving/behavior_planning -p use_sim_time:=True -p wheel_radius:=0.383 -p wheel_width:=0.235 -p wheel_base:=2.79 -p wheel_tread:=1.64 -p front_overhang:=1.0 -p rear_overhang:=1.1 -p left_overhang:=0.128 -p right_overhang:=0.128 -p vehicle_height:=2.5 -p max_steer_angle:=0.7'].

Expected behavior

I hope the ego vehicle to reach to the destination.

Actual behavior

But the vehicle stops in the middle of the road, and does not move at all.

Steps to reproduce

Here's the ros2bag file: [ros2bag] I used the lane2let map here, and the initial/goal point as below (excerpted from launch.log file):

# Initial pose
x: 120.057753, 
y: -8.898988, 
z: 0.034443
qx: -0.000467, 
qy: -0.000028, 
qz: -0.007445, 
qw: 0.999972
# Goal pose
x: 209.993347, 
y: -9.808371, 
z: 0.275307
qx: 0.000000, 
qy: 0.000000, 
qz: -0.007468, 
qw: 0.999972

Versions

autoware.universe: 7877192cb5700c1db15109dcd959785fc381951d
Carla: 0.9.15

Possible causes

No response

Additional context

No response

VRichardJP commented 1 month ago

Unfortunatelly, your logs just show the main thread crashed at a random place. Most likely the root cause of the crash comes from another thread.

If you are able to reproduce the issue, could you try to catch it with gdb? You just need to recompile with -CMAKE_BUILD_TYPE=RelWithDebInfo and add launch-prefix='gnome-terminal -- gdb -ex=r --args' at the end of <node_container ...> in the behavior_planning.launch.xml. When the program crashes in the new terminal, GDB should automatically switch to the problematic thread and you should be able to get a backtrace with bt

maxime-clem commented 1 month ago

I have not been able to reproduce the issue with the bag. I was also unable to get the map you showed. It looks different from the Town03 whose link you shared. Unless you are using a custom map_config.yaml ?

autowarefoundation / autoware.universe