uber-research / LaneGCN

[ECCV2020 Oral] Learning Lane Graph Representations for Motion Forecasting
https://arxiv.org/abs/2007.13732
Other
496 stars 131 forks source link

Fix bug with miss rate threshold in evaluation script #1

Closed wqi closed 3 years ago

wqi commented 3 years ago

Thanks for supporting open source research and for sharing your code with the community! Having a reference implementation is a huge benefit for reproducibility and I really appreciated being able to poke around the inside of LaneGCN.

However, while running the evaluation code in test.py I noticed that the computed miss rate was suspiciously low, although reported FDE/ADE values seemed to be in the right ballpark. These incongruent metrics seem to stem from a faulty configuration of the miss_threshold to 20m, instead of the Argoverse standard of 2m.

After changing the threshold, the computed miss rate of the pre-trained model is 0.16, which seems in line with expectaitons.

Created a PR to save folks some confusion in the future, but please let me know if I'm missing anything!

CLAassistant commented 3 years ago

CLA assistant check
All committers have signed the CLA.

chenyuntc commented 3 years ago

Thanks @wqi !