Confidence-Aware Learning for Deep Neural Networks

This repository provides the code for training with Correctness Ranking Loss presented in the paper "Confidence-Aware Learning for Deep Neural Networks" accepted to ICML2020.

Getting Started

Requirements

* ubuntu 18.0.4, cuda10
* python 3.6.8
* pytorch >= 1.2.0
* torchvision >= 0.4.0 

Datasets

• CIFAR-10, CIFAR-100, SVHN

How to Run

Arguments

Train with Correctness Ranking Loss

# Examples 
python main.py --save_path ./res_cifar10/softmax/ --model res --data cifar10 --rank_target softmax --rank_weight 1.0 --gpu 0 
python main.py --save_path ./vgg_cifar100/entropy/ --model vgg --data cifar100 --rank_target entropy --rank_weight 1.0 --gpu 0 

Train baseline models

• Set rank_weight = 0.

  # Examples
  python main.py --save_path ./res_cifar10/baseline/ --model res --data cifar10 --rank_weight 0.0 --gpu 0 
  python main.py --save_path ./vgg_cifar100/baseline/ --model vgg --data cifar100 --rank_weight 0.0 --gpu 0 

Evaluate the trained model

# Calculate performance measures from the trained model `file_name.pth` located in `save_path`

|---- test.py
|     |---- save_path
|           |---- file_name.pth
|           |---- result.log

python test.py --save_path ./res_cifar10/ --file_name model --model res --data cifar10 --gpu 0 
python test.py --save_path ./vgg_svhn/ --file_name model --model vgg --data svhn --gpu 0

Results

Performance measures

• Accuracy
• AURC, EAURC
• Expected Calibration Error(ECE)
• Negative Log Likelihood(NLL)
• Brier Score
• AUPR Error, FPR 95% TPR

Results on CIFAR-100

• More results can be found in the paper.

Citation

@inproceedings{moon2020crl,
  title={Confidence-Aware Learning for Deep Neural Networks},
  author={Moon, Jooyoung and Kim, Jihyo and Shin, Younghak and Hwang, Sangheum},
  booktitle={International Conference on Machine Learning},
  year={2020}
}

Contact for issues

• JooYoung Moon, answn3475@ds.seoultech.ac.kr