GzyAftermath
commented
1 year ago Hi. KD does not have to use logits information. Div loss between logits is the KD loss for logit-based methods only, while CAT loss is the KD loss for ours.
获取 Outlook for iOShttps://aka.ms/o0ukef
发件人: Mohammad Amin @.> 发送时间: Sunday, July 9, 2023 10:33:49 PM 收件人: GzyAftermath/CAT-KD @.> 抄送: Subscribed @.***> 主题: [GzyAftermath/CAT-KD] CAT-KD Loss (Issue #1)
Hi, Thanks for sharing the code. Amazing work btw!
I just have a question regarding the implementation of CAT-KD. As it's mentioned in the paper, and as the name suggests, this is supposed to be a combination of CAT loss and KD loss, right? But in the code, it seems that it's a combination of CAT and loss w.r.t GT labels (and not the teacher logits). https://github.com/GzyAftermath/CAT-KD/blob/4467073ca31b9929e7c45f5a6761e5e1b7c9442f/mdistiller/distillers/CAT_KD.py#L58-L67 Could you please clarify this?
Best.
― Reply to this email directly, view it on GitHubhttps://github.com/GzyAftermath/CAT-KD/issues/1, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYY4EH7Z3VCY4MRD423UP33XPK6M3ANCNFSM6AAAAAA2DRGUWA. You are receiving this because you are subscribed to this thread.Message ID: @.***>
m-parchami
Hi, Thanks for sharing the code. Amazing work btw!
I just have a question regarding the implementation of CAT-KD. As it's mentioned in the paper, and as the name suggests, this is supposed to be a combination of CAT loss and KD loss, right? But in the code, it seems that it's a combination of CAT and loss w.r.t GT labels (and not the teacher logits). https://github.com/GzyAftermath/CAT-KD/blob/4467073ca31b9929e7c45f5a6761e5e1b7c9442f/mdistiller/distillers/CAT_KD.py#L58-L67 Could you please clarify this?
Also one more thing, I couldn't find the temperatures that you have used for logit loss between teacher and student (e.g. in the case of KD) in the supplement. Is it possible that you could share those too?
All the best!