argoverse / argoverse-api

Official GitHub repository for Argoverse dataset
https://www.argoverse.org
Other
839 stars 240 forks source link

Added brier score for p-metrics in forecasting #209

Closed jagjeet-singh closed 3 years ago

jagjeet-singh commented 3 years ago

This PR adds brier score based metric to forecasting. The two new metrics are: brier-minFDE = (1.0-p)^2 + minFDE and brier-minADE = (1.0-p)^2 + minADE

Motivation for using brier score for assessing uncertainty:

image

Top methods from eval ai leaderboard: image

johnwlambert commented 3 years ago

Thanks for the PR, @jagjeet-singh. Would you mind adding a few sentences here explaining the motivation for the change, and links to any relative literature? Maybe those could go into the docstrings, as well?

Which scenarios do we have tested so far?