antoinetlc
commented
3 years ago Hello,
Thank you for your questions. I used distillation only for the categorical emotions and not for the valence and arousal values. I replaced the cross entropy loss for categorical emotions by a KL divergence term between the teacher and student predictions (after having applied a softmax, in order to get probability distributions). The part of the loss that deals with valence and arousal stayed the same (see paper). I did not use a temperature parameter.
From my experience, this worked great with the AffectNet dataset. However, it might be data dependent and there might be other ways of doing the distillation such as :
- keep the cross entropy between the student prediction and label coming from the dataset and add a KL divergence term between the student and teacher predictions as a regularization (in this case, add a coefficient in front of the KL divergence term, to control the amount of regularization you want)
- You could also add distillation for valence and arousal. In this case, I would use a L2 loss between the valence and arousal predictions of the teacher and student instead of the KL divergence as this is a regression problem (a KL divergence with negative values for regression will break...). Again this distillation term could either replace the original loss with the label coming from the dataset or come on top of it as a regularization
It is hard to know what will work better as it is data dependent. I would recommend you to try on a validation set and see what gives you the best accuracy there to select your best model.
Hope it helps!
segalinc
nlml
Hi, do you apply the KL divergence loss to both valence arousal and expression? Can you provide more details about it? For instance what do you pass to the loss? Do you create distributions from the prediction and draw a sample and pass them to the loss? digitize the predictions before passing them to the loss? pass the predictions as they are? Did you also use some temperature parameter? Thank you