Closed jagjeet-singh closed 3 years ago
Thanks for the PR, @jagjeet-singh. Would you mind adding a few sentences here explaining the motivation for the change, and links to any relative literature? Maybe those could go into the docstrings, as well?
Which scenarios do we have tested so far?
This PR adds brier score based metric to forecasting. The two new metrics are:
brier-minFDE = (1.0-p)^2 + minFDE
andbrier-minADE = (1.0-p)^2 + minADE
Motivation for using brier score for assessing uncertainty:
Top methods from eval ai leaderboard:![image](https://user-images.githubusercontent.com/11771909/115798376-1fe50980-a3a4-11eb-9ef2-5cdbfc1465b1.png)