Introduction

KAmalEngine (KAE) aims at building a lightweight algorithm package for Knowledge Amalgamation, Knowledge Distillation and Model Transferability Estimation.

Features:

• Algorithms for knowledge amalgamation and distillation
• Deep model transferability estimation based on attribution maps
• Predefined callbacks & metrics for evaluation and visualization
• Easy-to-use tools for multi-tasking training, e.g. synchronized transformation

Table of contents

• Introduction
• Table of contents
• Quick Start
• Algorithms
  • 1. Knowledge Amalgamation
  • 2. Knowledge Distillation
  • 3. Model Transferability
• Transferability Graph
• Authors

Quick Start

Please follow the instructions in QuickStart.md for the basic usage of KAE. More examples can be found in examples, including knowledge amalgamation, knowledge distillation, transferability and model slimming.

Algorithms

1. Knowledge Amalgamation

Amalgamating Knowledge towards Comprehensive Classification

[Amalgamating Knowledge towards Comprehensive Classification](https://ojs.aaai.org/index.php/AAAI/article/view/4165) (*AAAI 2019*) ``` @inproceedings{shen2019amalgamating, author={Shen, Chengchao and Wang, Xinchao and Song, Jie and Sun, Li and Song, Mingli}, title={Amalgamating Knowledge towards Comprehensive Classification}, booktitle={AAAI Conference on Artificial Intelligence (AAAI)}, pages={3068--3075}, year={2019} } ``` **Scripts can be found in [examples/amalgamation/layerwise_ka](examples/amalgamation/layerwise_ka)**

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation

[Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation](https://openaccess.thecvf.com/content_ICCV_2019/papers/Shen_Customizing_Student_Networks_From_Heterogeneous_Teachers_via_Adaptive_Knowledge_Amalgamation_ICCV_2019_paper.pdf) (*ICCV 2019*)

``` @inproceedings{shen2019customizing, title={Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation}, author={Shen, Chengchao and Xue, Mengqi and Wang, Xinchao and Song, Jie and Sun, Li and Song, Mingli}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages={3504--3513}, year={2019} } ``` **Scripts can be found in [examples/amalgamation/customize_ka](examples/amalgamation/customize_ka)**

Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

[Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More](https://openaccess.thecvf.com/content_CVPR_2019/papers/Ye_Student_Becoming_the_Master_Knowledge_Amalgamation_for_Joint_Scene_Parsing_CVPR_2019_paper.pdf) (*CVPR 2019*) ``` @inproceedings{ye2019student, title={Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More}, author={Ye, Jingwen and Ji, Yixin and Wang, Xinchao and Ou, Kairi and Tao, Dapeng and Song, Mingli}, booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, pages={2829--2838}, year={2019} } ```

Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning

[Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123510630.pdf) (*ECCV 2020*)

``` @inproceedings{luo2020collaboration, title={Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning}, author={Luo, Sihui and Pan, Wenwen and Wang, Xinchao and Wang, Dazhou and Tang, Haihong and Song, Mingli}, booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16}, pages={631--646}, year={2020}, organization={Springer} } ``` **Scripts can be found in [examples/knowledge_amalgamation/soka](examples/knowledge_amalgamation/soka)**

Knowledge Amalgamation for Object Detection With Transformers

[Knowledge Amalgamation for Object Detection With Transformers](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10091778) (*TIP 2021*) ``` @article{zhang2023knowledge, title={Knowledge Amalgamation for Object Detection With Transformers}, author={Zhang, Haofei and Mao, Feng and Xue, Mengqi and Fang, Gongfan and Feng, Zunlei and Song, Jie and Song, Mingli}, journal={IEEE Transactions on Image Processing}, volume={32}, pages={2093--2106}, year={2023}, publisher={IEEE} } ``` **Scripts can be found in [examples/amalgamation/transformer_ka](examples/amalgamation/transformer_ka)**

FedKA: Federated Selective Aggregation for Knowledge Amalgamation

[Federated Selective Aggregation for Knowledge Amalgamation](https://www.sciencedirect.com/science/article/pii/S2709472323000163) (*CHIP 2022*)

``` @article{XIE2023100053, title = {Federated selective aggregation for on-device knowledge amalgamation}, journal = {Chip}, volume = {2}, number = {3}, pages = {100053}, year = {2023}, issn = {2709-4723}, doi = {https://doi.org/10.1016/j.chip.2023.100053}, url = {https://www.sciencedirect.com/science/article/pii/S2709472323000163}, author = {Donglin Xie and Ruonan Yu and Gongfan Fang and Jiaqi Han and Jie Song and Zunlei Feng and Li Sun and Mingli Song} } ```

2. Knowledge Distillation

Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers

[Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://ojs.aaai.org/index.php/AAAI/article/view/6174) (*AAAI 2020*)

``` @inproceedings{zhao2020hearing, title={Hearing lips: Improving lip reading by distilling speech recognizers}, author={Zhao, Ya and Xu, Rui and Wang, Xinchao and Hou, Peng and Tang, Haihong and Song, Mingli}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={34}, number={04}, pages={6917--6924}, year={2020} } ``` **Scripts can be found in [examples/distillation/libs](examples/distillation/libs)**

Progressive Network Grafting for Few-Shot Knowledge Distillation

[Progressive Network Grafting for Few-Shot Knowledge Distillation](https://ojs.aaai.org/index.php/AAAI/article/view/16356) (*AAAI 2021*)

``` @inproceedings{shen2021progressive, title={Progressive network grafting for few-shot knowledge distillation}, author={Shen, Chengchao and Wang, Xinchao and Yin, Youtan and Song, Jie and Luo, Sihui and Song, Mingli}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={35}, number={3}, pages={2541--2549}, year={2021} } ``` **Scripts can be found in [/examples/distillation/graft_kd](/examples/distillation/graft_kd)**

KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation

[KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation](https://www.ijcai.org/proceedings/2021/0444.pdf) (*IJCAI 2021*)

``` @inproceedings{ijcai2021p444, title = {KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation}, author = {Xue, Mengqi and Song, Jie and Wang, Xinchao and Chen, Ying and Wang, Xingen and Song, Mingli}, booktitle = {Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, {IJCAI-21}}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, editor = {Zhi-Hua Zhou}, pages = {3228--3234}, year = {2021}, month = {8}, note = {Main Track}, doi = {10.24963/ijcai.2021/444}, url = {https://doi.org/10.24963/ijcai.2021/444}, } ``` **Scripts can be found in [/examples/distillation/kd_explainer](/examples/distillation/kd_eplainer)**

CMI: Contrastive Model Inversion for Data-Free Knowledge Distillation

[Contrastive Model Inversion for Data-Free Knowledge Distillation](https://www.ijcai.org/proceedings/2021/0327.pdf) (*IJCAI 2021*)

``` @inproceedings{ijcai2021p327, title = {Contrastive Model Invertion for Data-Free Knolwedge Distillation}, author = {Fang, Gongfan and Song, Jie and Wang, Xinchao and Shen, Chengchao and Wang, Xingen and Song, Mingli}, booktitle = {Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, {IJCAI-21}}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, editor = {Zhi-Hua Zhou}, pages = {2374--2380}, year = {2021}, month = {8}, note = {Main Track}, doi = {10.24963/ijcai.2021/327}, url = {https://doi.org/10.24963/ijcai.2021/327}, } ``` **Scripts can be found in [examples/distillation/cmi](examples/distillation/cmi)**

MosaicKD: Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

[Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://proceedings.neurips.cc/paper_files/paper/2021/file/63dc7ed1010d3c3b8269faf0ba7491d4-Paper.pdf) (*NeurIPS 2021*)

``` @article{fang2021mosaicking, title={Mosaicking to distill: Knowledge distillation from out-of-domain data}, author={Fang, Gongfan and Bao, Yifan and Song, Jie and Wang, Xinchao and Xie, Donglin and Shen, Chengchao and Song, Mingli}, journal={Advances in Neural Information Processing Systems}, volume={34}, pages={11920--11932}, year={2021} } ```

FastDFKD: Up to 100× Faster Data-free Knowledge Distillation

[Up to 100× Faster Data-free Knowledge Distillation](https://ojs.aaai.org/index.php/AAAI/article/view/20613) (*AAAI 2022*)

``` @inproceedings{fang2022up, title={Up to 100x faster data-free knowledge distillation}, author={Fang, Gongfan and Mo, Kanya and Wang, Xinchao and Song, Jie and Bei, Shitao and Zhang, Haofei and Song, Mingli}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={36}, number={6}, pages={6597--6604}, year={2022} } ```

Data-Free Adversarial Distillation

[Data-Free Adversarial Distillation](https://arxiv.org/abs/1912.11006)

``` @article{fang2019data, title={Data-free adversarial distillation}, author={Fang, Gongfan and Song, Jie and Shen, Chengchao and Wang, Xinchao and Chen, Da and Song, Mingli}, journal={arXiv preprint arXiv:1912.11006}, year={2019} } ``` **Scripts can be found in [examples/distillation/dfkd](examples/distillation/dfkd)**

Safe Distillation Box

[Safe Distillation Box](https://ojs.aaai.org/index.php/AAAI/article/view/20219) (*AAAI 2022*)

``` @inproceedings{ye2022safe, title={Safe distillation box}, author={Ye, Jingwen and Mao, Yining and Song, Jie and Wang, Xinchao and Jin, Cheng and Song, Mingli}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={36}, number={3}, pages={3117--3124}, year={2022} } ``` **Scripts can be found in [examples/distillation/sdb](examples/distillation/sdb)**

Spot-adaptive Knowledge Distillation

[Spot-adaptive Knowledge Distillation](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9767610) (*TIP 2022*)

``` @article{song2022spot, title={Spot-adaptive knowledge distillation}, author={Song, Jie and Chen, Ying and Ye, Jingwen and Song, Mingli}, journal={IEEE Transactions on Image Processing}, volume={31}, pages={3359--3370}, year={2022}, publisher={IEEE} } ``` **Scripts can be found in [examples/distillation/sakd](examples/distillation/sakd)**

Tree-like Decision Distillation

[Tree-like Decision Distillation](https://openaccess.thecvf.com/content/CVPR2021/papers/Song_Tree-Like_Decision_Distillation_CVPR_2021_paper.pdf) (*CVPR 2021*)

``` @inproceedings{song2021tree, title={Tree-like decision distillation}, author={Song, Jie and Zhang, Haofei and Wang, Xinchao and Xue, Mengqi and Chen, Ying and Sun, Li and Tao, Dacheng and Song, Mingli}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={13488--13497}, year={2021} } ``` **Script can be found in [examples/distillation/tdd](./examples/distillation/tdd)**

Context Correlation Distillation for Lip Reading

[Context Correlation Distillation for Lip Reading](https://www.jcad.cn/cn/article/doi/10.3724/SP.J.1089.2022.19723) (*计算机辅助设计与图形学学报*)

``` @articleInfo{19723, title = "针对唇语识别的上下文相关性蒸馏方法", journal = "计算机辅助设计与图形学学报", volume = "34", number = "19723, pages = "1559", year = "2022", note = "", issn = "1003-9775", doi = "10.3724/SP.J.1089.2022.19723", url = "https://www.jcad.cn/article/doi/10.3724/SP.J.1089.2022.19723", author = "赵雅","冯尊磊","王慧琼","宋明黎",keywords = "唇语识别","知识蒸馏","跨模态", } ``` **Scripts can be found in [examples/distillation/c2kd](examples/distillation/c2kd)**

3. Model Transferability

Deep model transferability from attribution maps

[Deep model transferability from attribution maps](https://proceedings.neurips.cc/paper_files/paper/2019/file/e94fe9ac8dc10dd8b9a239e6abee2848-Paper.pdf) (*NeurIPS 2019*)

``` @inproceedings{song2019deep, title={Deep model transferability from attribution maps}, author={Song, Jie and Chen, Yixin and Wang, Xinchao and Shen, Chengchao and Song, Mingli}, booktitle={Advances in Neural Information Processing Systems}, pages={6182--6192}, year={2019} } ```

DEPARA: Deep Attribution Graph for Deep Knowledge Transferability

[DEPARA: Deep Attribution Graph for Deep Knowledge Transferability](https://openaccess.thecvf.com/content_CVPR_2020/papers/Song_DEPARA_Deep_Attribution_Graph_for_Deep_Knowledge_Transferability_CVPR_2020_paper.pdf) (*CVPR 2020*)

``` @inproceedings{song2020depara, title={DEPARA: Deep Attribution Graph for Deep Knowledge Transferability}, author={Song, Jie and Chen, Yixin and Ye, Jingwen and Wang, Xinchao and Shen, Chengchao and Mao, Feng and Song, Mingli}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={3922--3930}, year={2020} } ```

Transferability Graph

This is an example for deep model transferability on 300 classification models. see examples/transferability for more details.

Authors

This project is developed by VIPA Lab from Zhejiang University and Zhejiang Lab