search

huawei-noah / SMARTS

Scalable Multi-Agent RL Training School for Autonomous Driving
MIT License
925 stars 186 forks source link

Abruptly breaking waypoints affect vehicle_gap metric #2018

Closed Adaickalavan closed 1 year ago

Adaickalavan commented 1 year ago

High Level Description

Waypoints tend to break abruptly on straight road with junctions.

Version

1.1.0

Steps to reproduce the bug

Replay latest t3/training/merge_exit_sumo_t_agents_1 scenario with an endless mission ego agent and observe the waypoints.

System info

No response

Error logs and screenshots

The following is a sequence of ordered images in time. Each image comprise of a map and its corresponding rgb observation. Each rgb observation depicts 3 consecutive time points, with the latest being on the left.

t1

t3

t4

t6

t7

Impact (If known)

The vehicle_gap metrics uses waypoints to traverse across different road segments in order to identify whether the vehicle of interest is in the same lane as the ego. When waypoints abruptly get truncated (except at the end of the map), this curtails the ability to reliably check whether the ego is properly following the vehicle of interest.

Either we fix this abruptly ending waypoints or change the method used to identify lanes across different road segments in vehicle_gap metric.

saulfield commented 1 year ago

Hi @Adaickalavan, I tried to reproduce this using the latest merge_exit_sumo_t_agents_1 scenario and the current master branch, but can't seem to get those large jumps you're seeing, though it's a bit hard to tell how big they are from the images. I took some quick stats of the distance between the ego and the last waypoint in each path for the straight section:

n=805 min=27.598640610308706, max=35.14912621280988
mean=31.720287135398063 stdev=0.7905507692617167

Most distances on the straight part are close to the default lookahead value of 32, though there was one junction that jumped down to 27.6. I added a small fix to help with junction detection which seemed to improve it a bit:

n=800 min=29.90648407709022, max=33.42882447941028
mean=31.705357775518344 stdev=0.6368442742946877

The fix is on saul/waypoints-issue-2018 if you want to try it out.

These values seem reasonable to me, especially for a map with lots of overlapping junctions lanes. Some noise is to be expected. Also consider increasing the lookahead value for the waypoints sensor if you need to guarantee a larger range.

Adaickalavan commented 1 year ago

Similar abruptly terminating waypoints is also seen in Argoverse map, namely, t3/training/ffd10ec2-715b-48af-a89d-b11f79927f63_agents_1. Here, the collision done criteria was set to false temporarily just to capture the images.

Instance A: t_1

Instance A: t_2

Instance B: t_1

Instance B: t_2