search

valeoai / ConfidNet

Addressing Failure Prediction by Learning Model Confidence
Other
163 stars 35 forks source link

How to choose which epoch to use? #4

Closed YingdaXia closed 4 years ago

YingdaXia commented 4 years ago

Hi authors,

Great work! I really appreciate you can release code of your Neurips paper.

One question is that how do you select which epoch to use (for next step training or report final results). I trained a baseline for Cifar10 and find that the results varies from epoch to epoch. Do you have any specific criterions?

Thanks!

chcorbi commented 4 years ago

Hi YingdaXia,

Regarding the original classifier model, best epoch is selected on validation set accuracy. MCP, MCDropout and TrustScore results are all computed using this same baseline.

From that model, we attach ConfidNet layers and train them. We select best epoch for ConfidNet based on AUPR-Error on validation dataset and report results for that epoch.

More details about the implementation are available also on the NeurIPS paper's supplemental: https://papers.nips.cc/paper/8556-addressing-failure-prediction-by-learning-model-confidence

Best, Charles

YingdaXia commented 4 years ago

Thank you so much for your explanations!

Best, Yingda