Learning to Defer to a Population: A Meta-Learning Approach

This is a PyTorch implementation of the following paper:

Learning to Defer to a Population: A Meta-Learning Approach Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, Eric Nalisnick 27th International Conference on Artificial Intelligence and Statistics (AISTATS 2014)

Environment setup

To create a conda environment l2d with all necessary dependencies run: conda env create -f environment.yml or use the following explicit instructions:

conda create --name l2d python=3.9
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
conda install numpy scipy matplotlib jupyterlab jupyter_console jupyter_client scikit-learn
pip install attrdict

Usage

To reproduce figure 3 on varying population diversity on image classification tasks:

• DATASET {gtsrb/cifar10/ham10000}
  • gtsrb is traffic sign detection; cifar10 is image recognition, and ham10000 is skin lesion diagnosis
  • For ham10000 follow the instructions in /data/HAM10000/README.md to setup dataset
• L2D {single/pop}
  • single is "single-L2D" (which also runs "L2D-Pop (finetune)", and pop is "L2D-Pop (NP)"
• EXPERT_OVERLAP_PROB {0.1/0.2/0.4/0.6/0.8/0.95}
  • This controls the expert overlap probability varying from specialized experts (p=0.1) to near-identical experts (p=0.95)
• SEED

In the case of cifar10 and ham10000, the networks are warmstarted and so we first need to train a stand-alone classifier: python train_classifier.py --seed=[SEED] --dataset=[DATASET]

Then run bash train_[DATASET].sh [L2D] [EXPERT_OVERLAP_PROB] train [SEED]

To reproduce figure 4 on CIFAR-20 which also has an additional method using conditional neural process with attention mechanism:

• L2D {single/pop/pop_attn}
  • pop_attn is "L2D-Pop (NP+attention)"
• EXPERT_OVERLAP_PROB {0.1/0.2/0.4/0.6/0.8/0.95/1.0}
  • This experiment evaluates on an additional setting p=1.0

Again we need to pretrain a stand-alone classifier: python train_classifier.py --seed=[SEED] --dataset=cifar20_100

Then run: bash train_cifar20_100.sh [L2D] [EXPERT_OVERLAP_PROB] train [SEED]

Acknowledgements

This codebase is largely an extension of the codebases of OvA-L2D [Verma & Nalisnick] and learn-to-defer [Mozannar & Sontag]. We also acknowledge code related to attention mechanism from TNP-pytorch [Nguyen & Grover] and bnp [Lee et. al.].

Troubleshooting

Please open an issue in this repository or contact Dharmesh.

Citation

Please consider citing our conference paper

@inproceedings{tailor2024learning,
  title           = {{Learning to Defer to a Population: A Meta-Learning Approach}},
  booktitle       = {Proceedings of the 27th International Conference on Artificial Intelligence and Statistics},
  author          = {Tailor, Dharmesh and Patra, Aditya and Verma, Rajeev and Manggala, Putra and Nalisnick, Eric},
  year            = {2024}
}