search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.58k stars 600 forks source link

[WOSAC] 2023 V1 leaderboard == 2024 leaderboard? #817

Closed pengzhenghao closed 1 month ago

pengzhenghao commented 2 months ago

Hi, I want to double check if the 2023 upgraded v1 version of the WOSAC is the same as the latest 2024 leaderboard. This confuses me cuz I saw MVTE has drastically different result in two leaderboards.

nicomon24 commented 2 months ago

Hi @pengzhenghao, no this is a new configuration of the metrics as explicitly called out on the website (section "[Update March 2024]" under leaderboard). Specifically, you can see that the metrics_config changed (which accounts for most of the changes in scores, old config)

Sorry about the confusion, the v0->v1 change was prior to the new challenge and with a different naming scheme, hopefully with this year-based one it should be clearer.