search

dkozlov / awesome-knowledge-distillation

Awesome Knowledge Distillation
Apache License 2.0
3.43k stars 493 forks source link

Please add a relevant paper in CVPR 2024 #38

Open sunshangquan opened 5 months ago

sunshangquan commented 5 months ago

Hi @dkozlov I notice that there is no paper in 2024 listed yet. Could you please add a relevant paper from CVPR 2024 Highlight. The information is given below:

Title: Logit Standardization in Knowledge Distillation Paper: https://arxiv.org/abs/2403.01427 Github: https://github.com/sunshangquan/logit-standardization-KD Supplements: https://sunsean21.github.io/resources/cvpr2024_supp.pdf

The paper discusses the possibility of assigning temperatures distinctly between teacher/student and dynamically across samples. It then proposes a weighted Z-score logit standardization as a plug-and-play preprocess, capable of boosting the existing logit-based KD methods. Thank you for your attention.

dkozlov commented 3 months ago

Hi @sunshangquan,

Sorry for the late reply. Could you please create a new pull request?

sunshangquan commented 3 months ago

Hi @dkozlov , I have created a pull request. Thank you for your attention.