What is the difference between Born Again Network and your self-training KD method?

yuanli2333 / Teacher-free-Knowledge-Distillation

Knowledge Distillation: CVPR2020 Oral, Revisiting Knowledge Distillation via Label Smoothing Regularization

MIT License

580 stars 67 forks source link

What is the difference between Born Again Network and your self-training KD method? #18

Closed JiyueWang closed 4 years ago

yuanli2333 commented 4 years ago

Yes, the first teacher-free is the same as BAN, the second one is different. The analysis on the relationship between LRS and KD can reveal why teaching itself also work.

JiyueWang commented 4 years ago

Thanks for your confirmation. The analysis in Section 2 and 3 is excellent.