Define Test Procedures to check Simulator's Compatibility with Autoware

[sglee-morai]

Description

To define test procedures (e.g. test codes) to check whether a certain simulator is compatible with Autoware

Needs to define 1) what feature of Autoware should be able to integrate with a certain simulator 2) requirement for the simulator (e.g. a simulator should receive autoware_auto_msgs/VehicleControlCommand and the vehicle model within the simulator should respond accordingly.) 3) test cases for the requirements 4) test codes

Purpose

To ensure that any simulator is compatible with the latest version of Autoware Ultimately, streamlining the development process of Autoware-compatible simulators

Possible approaches

A very simple method to check the compatibility of MORAI SIM: Drive with Autoware is included in this documentation, which can give a very brief idea at least. Link

Discussion needed: how to automate the tests or at least do them in a systematic way.

[sglee-morai]

Proceed in two steps:

1) creating (mostly) manual procedures 2) automate the procedures

Coverage of the test

1) [Initial] cover the interface between the simulator and Autoware 2) [Future] test would cover other requirements defined in the documentation (related discussion link)

[sglee-morai]

Test Procedures to test interfaces between a simulator and Autoware

Requirement overview

link for the interface requirement: link

Related Repository

(WIP)

Detailed Test Procedure

(WIP)

[sglee-morai]

Discussion / Questions

where to put the test codes

• autoware.universe/

location of the msg definition

confused

• gitlab autoware_auto_msgs repo
• github autoware_msgs repo

gitlab

• autoware_auto_vehicle_msgs/TurnIndicatorsCommand
• autoware_auto_vehicle_msgs/HazardLightsCommand
• autoware_auto_control_msgs/AckermannControlCommand
• autoware_auto_vehicle_msgs/GearCommand
• autoware_auto_vehicle_msgs/GearReport
• autoware_auto_vehicle_msgs/TurnIndicatorsReport
• autoware_auto_vehicle_msgs/HazardLightsReport

unable to find

• tier4_vehicle_msgs/msg/VehicleEmergencyStamped
• autoware_auto_vehicle_msgs/ControlModeReport
• autoware_auto_vehicle_msgs/SteeringReport
• autoware_auto_vehicle_msgs/VehicleReport

https://github.com/autowarefoundation/autoware/blob/main/autoware.repos

https://github.com/tier4/autoware_auto_msgs

creating a specific test map for test automation

how to check frequency & how important it is

use of unit testing framework: python unittest

[sglee-morai]

Test Procedure Draft (WIP)

1) ControlModeCommand & ControlModeReport

• Send ControlModeCommand with the command values defined in ControlModeCommand_Constants
• Receive ControlModeReport
• Check whether the received value is the same as the value sent as the command

2) GearCommand & Report

Case 1-4)

• Send GearCommand: [P, N, R, D]
• Received GearReport: [P, N, R, D]
• Check whether the gear is changed according to the command

3) AckermannControlCommand-LongitudinalCommand & VelocityReport

Case 1)

• Send AckermannControlCommand with longitudinal.speed value greater than 0
• Receive VelocityReport
• Check whether lonitudinal_velocity is greater than 0

Case 2)

• Send AckermannControlCommand with longitudinal.acceleration value greater than 0
• Receive VelocityReport
• Check whether lonitudinal_velocity is greater than 0

4) AckermannControlCommand-AckermannLateralCommand & SteeringReport

• Send AckermannControlCommand with steering_tire_angle value greater than 0 [less than 0]
• Receive SteeringReport
• Check whether steering_tire_angle is close to the command value (% error should be less than (TBD))

5) TurnIndicatorsCommand & TurnIndicatorsReport

• Send TurnIndicatorsCommand with the command values defined in TurnIndicatorsCommand_Constants
• Receive TurnIndicatorsReport
• Check whether the received value is the same as the value sent as the command
• [Manual Procedure] Check whether the turn indicator in the simulator vehicle model turns on/off accordingly

6) HazardLightsCommand & HazardLightsReport

• Send HazardLightsCommand with the command values defined in TurnIndicatorsCommand_Constants
• Receive HazardLightsReport
• Check whether the received value is the same as the value sent as the command
• [Manual Procedure] Check whether the hazard light in the simulator vehicle model turns on/off accordingly

7) EmergencyState

[TBD]

[sglee-morai]

Update 22-05-25

• The initial version of the test codes for No. 1, 2, 3 is written

Plan

• Create PR for what's written for now, and then work on the others.
  • Feedback from Tsutake-san would be very helpful, since I'm hoping that the code can be used for TierIV SIM as well

[sglee-morai]

Update 22-06-08

Test code for No.1 to 6 is written (Already some automation features are already implemented)

PR has been delayed since internally we have found things to refactor first, which Soohyeok (MORAI) is doing some refactoring.

[WJaworskiRobotec]

PR with the code created here : link

[sglee-morai]

Update 22-07-06

Want to ask for a review: TierIV to test run the code. However, I have a question regarding this, though. When a certain person (MORAI in this case) asks other members to run the code in their machine, how he or she can do that?

For me to have Soohyuk's code in my machine, I had to do the followings

• open terminal
• change working directory to ~/autoware/src/universe
• change remote to Soohyuk's Autoware.Universe repository (which is a fork from the original Autoware.Universe repository)
• git pull
• git checkout [Soohyuk's latest commit]

Is this a typical process for other members as well in case wanting to get the code and run it in his/her local machine?

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity.