Closed shirleyuw closed 6 years ago
@satyanshukla , please help review this one as it is relevant to your next project. Nhat has also implemented/tested Concrete Dropout (see this closed PR) so you may want to check his code or pick his brain if you have questions.
Let's wait to have some test first, to make sure that the implementation is correct.
Here are a few tests:
I will run these tests. Ken, can you explain more on the last test? Thanks, Shirley
On Jun 22, 2018, at 12:01 PM, Kenneth Tran notifications@github.com wrote:
Let's wait to have some test first, to make sure that the implementation is correct.
Here are a few tests:
Calibration test (mentioned in Yarin's paper) Do we get better match in the calibration test than before concrete dropout? Does the drop rate converge to 0 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
A few changes: (1) Implemented concrete dropout class, such that an optimal dropout rate can be automatically learnt based on the 'Concrete dropout' paper. For concrete dropout, predictions with uncertainty can be produced without specifying a dropout rate. I still kept the MCDropout class with a user-provided dropout rate. (2) Implemented two metrics to evaluate uncertainty: (a) Mean Scaled Inverse Score (MSIS) used in M6 time series competition for uncertainty, and (b) Coverage of prediction intervals, e.g. does the 95% prediction interval really have 95% chance to include the observations in test set? (3) demo the concrete dropout and metrics for uncertainty in the tutorials/incorporating-uncertainty.ipynt