autowarefoundation / autoware

Autoware - the world's leading open-source software project for autonomous driving
https://www.autoware.org/
Apache License 2.0
8.58k stars 2.88k forks source link

Autoware weekly scenario tests are failing. #4752

Open mitsudome-r opened 1 month ago

mitsudome-r commented 1 month ago

Checklist

Description

The scenarios in AWF weekly tests are failing. https://evaluation.ci.tier4.jp/evaluation/reports/ca72c235-6701-53e4-9b9a-1c9c256ef413?project_id=awf

After downloading the log file, I found out that control node container is crashing.

[openscenario_interpreter_node-3] [ERROR] [component_container_mt-36]: process has died [pid 17082, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=control_container -r __ns:=/control -p use_sim_time:=True -p wheel_radius:=0.383 -p wheel_width:=0.235 -p wheel_base:=2.79 -p wheel_tread:=1.64 -p front_overhang:=1.0 -p rear_overhang:=1.1 -p left_overhang:=0.128 -p right_overhang:=0.128 -p vehicle_height:=2.5 -p max_steer_angle:=0.7'].

Expected behavior

Control component container does not crash.

Actual behavior

Control component container crashes.

Steps to reproduce

Build Autoware and run scenarios. Here's the repos file from the evaluator 1f8d7e07-0b16-4cd7-aa4c-f80c2a748eca_autoware.proj.repos.exact.txt.

Versions

No response

Possible causes

No response

Additional context

No response

mitsudome-r commented 1 month ago

I've also tested with the latest main branch today and still saw the same issue.

Steps to reproduce:

  1. Build Autoware and scenario simulator
  2. In terminal 1, run ros2 launch autoware_launch planning_simulator.launch.xml map_path:=$HOME/autoware_map/sample-map-planning vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit scenario_simulation:=true
  3. In terminal 2, run ros2 launch scenario_test_runner scenario_test_runner.launch.py architecture_type:=awf/universe record:=false scenario:='$(find-pkg-share scenario_test_runner)/scenario/sample.yaml' sensor_model:=sample_sensor_kit vehicle_model:=sample_vehicle initialize_duration:=90 launch_autoware:=false

I got the following error output from aeb module

[component_container_mt-36] PC: @                0x0 (unknown)
[component_container_mt-36] *** SIGSEGV (@0x0) received by PID 154061 (TID 0x7f0f06ffd640) from PID 0; stack trace: ***
[component_container_mt-36]     @     0x7f0f162fc4d6 google::(anonymous namespace)::FailureSignalHandler()
[component_container_mt-36]     @     0x7f0f1a642520 (unknown)
[component_container_mt-36]     @     0x7f0f1af3847d rclcpp::Time::Time()
[component_container_mt-36]     @     0x7f0eafd608f4 autoware::motion::control::autonomous_emergency_braking::AEB::onImu()
[component_container_mt-36]     @     0x7f0eafd6487a autoware::motion::control::autonomous_emergency_braking::AEB::fetchLatestData()
[component_container_mt-36]     @     0x7f0eafd69c83 autoware::motion::control::autonomous_emergency_braking::AEB::checkCollision()
[component_container_mt-36]     @     0x7f0eafd6a1ed autoware::motion::control::autonomous_emergency_braking::AEB::onCheckCollision()
[component_container_mt-36]     @     0x7f0eafdd968e diagnostic_updater::Updater::update()
[component_container_mt-36]     @     0x7f0eafd6e3f5 rclcpp::GenericTimer<>::execute_callback()
[component_container_mt-36]     @     0x7f0f1aec3f61 rclcpp::Executor::execute_any_executable()
[component_container_mt-36]     @     0x7f0f1aecb27a rclcpp::executors::MultiThreadedExecutor::run()
[component_container_mt-36]     @     0x7f0f1aadc253 (unknown)
[component_container_mt-36]     @     0x7f0f1a694ac3 (unknown)
[component_container_mt-36]     @     0x7f0f1a726850 (unknown)
mitsudome-r commented 1 month ago

After reverting https://github.com/autowarefoundation/autoware.universe/pull/6997 locally, it stopped failing, but I couldn't really trace the root cause yet. @soblin Could you think of any cause by any chance?