AaronHeee / MEAL

Official Implementation of MEAL: Multi-Model Ensemble via Adversarial Learning on AAAI 2019
177 stars 46 forks source link

MEAL: Multi-Model Ensemble via Adversarial Learning

This is the official PyTorch implementation for paper:

MEAL: Multi-Model Ensemble via Adversarial Learning (AAAI 2019, Oral).

Zhiqiang Shen*, Zhankui He*, Xiangyang Xue.

The key idea of this work is distilling diverse knowledge from different trained models (teachers) into a single student network, in order to learn an ensemble of multiple models without incurring additional testing costs. We use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously.

The student and teacher networks we implemented are listed in \models, and it is also easy to add new networks in our repo. The corresponding author of this paper is: Dr. Zhiqiang Shen.

If you find this helps your research, please cite:

@inproceedings{shen2019MEAL,
    title = {MEAL: Multi-Model Ensemble via Adversarial Learning},
    author = {Shen, Zhiqiang and He, Zhankui and Xue, Xiangyang},
    booktitle = {AAAI},
    year = {2019}
}

Quick Start

Environment

Python 3.6+

PyTorch 0.40+

Numpy 1.12+

Learning rate adjustment

I manually change the lr during training:

The factor a varies with number of teacher networks, between 1 and 2.

ImageNet model

Our trained ResNet-50 (the accuracy is even comparable to PyTorch official ResNet-152):

Models Top-1 error (%) Top-5 error (%) URL
ResNet-50 23.85 7.13 -
ResNet-101 22.63 6.44 -
ResNet-152 21.69 5.94 -
Our ResNet-50 21.79 5.99 Download (102.5M)