search

liuzechun / Nonuniform-to-Uniform-Quantization

Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation. In CVPR 2022.
115 stars 12 forks source link

Concerns about the loss function #4

Open HaoKun-Li opened 2 years ago

HaoKun-Li commented 2 years ago

Hello, thanks for your excellent work and code!

In the paper, the authors claim that they use the same knowledge distillation scheme as LSQ to train the quantized models. I show the screenshot as follows: image

However, in the paper of LSQ, LSQ uses both the distillation loss function of Hinton et al. (2015) with temperature of 1 and equal weight given to the standard loss during training. I show the screenshot as follows: image

When I read the code in this github. I notice that you have defined both the KD_loss and CrossEntropyLabelSmooth loss, but you use only distillation loss to train the quantized models. Is this a mistake, or a trick to improve the accuracy?

szq0214 commented 1 year ago

Hi @HaoKun-Li Thanks for your interest in this work, the KD loss here is following MEAL V2 without the standard loss term, rather than LSQ scheme. It can provide better results.