OpenDriveLab / UniAD

[CVPR'23 Best Paper Award] Planning-oriented Autonomous Driving
Apache License 2.0
3.11k stars 335 forks source link

Add option to use legacy planning metric definition #173

Closed wljungbergh closed 1 week ago

wljungbergh commented 3 months ago

As described in #166, there is a discrepancy between how planning metrics (L2 and collision rate) are computed in this repo vs previous works, such as VAD and ST-P3.

However, in the paper, these values are compared in the same table, leading to confusion. This PR enables computation of the UniAD metrics using the legacy definition used by VAD/ST-P3, making comparison easier. However, it still defaults to the "original" UniAD way of evaluating. This results in a ranking shift when comparing the methods, see the table below

Method L2 1s (m) L2 2s (m) L2 3s (m) CR 1s (%) CR 2s (%) CR 3s (%)
ST-P3 1.33 2.11 2.90 0.23 0.62 1.27
UniAD (your current metric) 0.51 0.98 1.65 0.10 0.15 0.61
UniAD (legacy ST-P3 metric) 0.42 0.64 0.91 0.07 0.10 0.22
VAD-Tiny 0.46 0.76 1.12 0.21 0.35 0.58
VAD-Base 0.41 0.70 1.05 0.07 0.17 0.41