Design of Autoware + AWSIM evaluation procedure

WJaworskiRobotec commented 9 months ago

Description

To create a design of tests for evaluation of Autoware stack with AWSIM. The validation process should be defined in more structured way, based on the previously performed tests:

Purpose

To make AWSIM widely used in development of Autoware

Definition of Done

[ ] Tests scenario description prepared (OpenScenario scenarios - to get the instruction from Piotr Zyskowski, to add the instruction to the autoware-documentation website)
[ ] Definition of success / failure (to deeply analyze failed scenarios)
[ ] Template of test results prepared (output from each experiments set)
[ ] Suggestion of how to handle encountered bugs (how to split them between AWSIM and Autoware etc.)

lchojnack commented 9 months ago

I've prepared a wiki page where I've started to describe how Autoware AWSIM verification can be done.

For test execution, I think we can use scenario_simulator_v2. As I know, the support for executing scenarios using AWSIM is in progress and will be available in the near future.

In order to think about and write down scenarios/test cases, I've prepared a spreadsheet. At the beginning we can try to reproduce the issues encountered in the previously performed tests (in the description)

Please let me know your feedback

lchojnack commented 7 months ago

I've started to prepare the AWSIM simulation which supports scenario_simulator_v2 scenarios.
As a first step, I've used the ITRI scenario from here

I've edited the scenario in the following way

@@ -31,6 +31,7 @@ OpenSCENARIO:
         Vehicle:
           name: ''
           vehicleCategory: car
+          model3d: "lexus_rx450h"
           BoundingBox:
             Center:
               x: 1.48
@@ -70,6 +71,7 @@ OpenSCENARIO:
         Vehicle:
           name: ''
           vehicleCategory: car
+          model3d: "taxi"
           BoundingBox:
             Center:
               x: 0

Then, I prepared the AWSIM with ITRI scene and ss2 support, and I ran the AWSIM with the scenario. The result is presented in the video below

https://github.com/autowarefoundation/autoware-projects/assets/8416175/9613682b-a646-475e-be14-938f05798dc6

At this moment, SS2 is able to spawn EGO and NPC and start the scenario execution. There is an issue that EGO is not able to detect the NPC and crash it. For that reason, the scenario fails. I'm investigating why the issue happens.

Tested with
AWSIM	https://github.com/tier4/AWSIM/commit/df76d6f1d7cca89e834007bd5aa459ac0469cdf2
Unitypackage	`ITRI_Campus_r02_HDRP.unitypackage`
Autoware	https://github.com/autowarefoundation/autoware/tree/2023.10
SS2	https://github.com/tier4/scenario_simulator_v2/commit/e187b8f7844c0c4464b3feff5680d66d8bbfde03

lchojnack commented 6 months ago

I've retested the scenario with an adjusted ITRI map to ll2. The scenario passes successfully. In the next steps, I will test the prepared AWSIM binary with other scenarios.

https://github.com/autowarefoundation/autoware-projects/assets/8416175/be8ce7c7-5fe6-4f3b-8909-05249a2d3211

I've also checked if the issue mentioned here still happens and it seems that is solved, so EGO is not driving below ll2 according to Autoware

https://github.com/autowarefoundation/autoware-projects/assets/8416175/c5cb7c93-43be-4326-945c-189141b0cf12

lchojnack commented 5 months ago

I've prepared the AWSIM binary of ITRI scene which supports ss2 and modified openscenarios. There are available here

lchojnack commented 4 months ago

I've started to work on the testing scenarios. I've prepared an initial version of scenarios that verify the following planning features:

So far, the scenarios are available on gdrive, but I think storing the scenarios in the GitHub repository could be a good idea.

Lane Change

scenario

https://github.com/autowarefoundation/autoware-projects/assets/8416175/a59ac096-e4db-487c-8088-0f08496ab397

Obstacle Avoidance

scenario

https://github.com/autowarefoundation/autoware-projects/assets/8416175/fcdd6381-4b7d-4d37-862f-85c97dfdbe68

lchojnack commented 3 months ago

I've created one more scenario when a pedestrian walks on a crosswalk.

PedestrianOnCrosswalk

scenario

https://github.com/autowarefoundation/autoware-projects/assets/8416175/b312bb27-eda9-45ea-aa31-72bfd0841e77

Conclusion

I've noticed that when the scenario is run with AWSIM, the EGO doesn't detect the NPC correctly and finally hits it. When the scenario is run only with SS2, then the EGO stops before the pedestrian. I think the issue is related to Autoware perception pipeline.

https://github.com/autowarefoundation/autoware-projects/assets/8416175/3b3d5412-82cc-47bf-ba4e-e10995f5a1ca

autowarefoundation / autoware-projects