search

alinlab / LfF

Learning from Failure: Training Debiased Classifier from Biased Classifier (NeurIPS 2020)
89 stars 9 forks source link

GCE implementation #3

Open sunniesuhyoung opened 3 years ago

sunniesuhyoung commented 3 years ago

Why is GCE implemented as

F.cross_entropy(logits, targets, reduction='none') * (Yg.squeeze().detach()**self.q)*self.q

instead of (1 - Yg.squeeze().detach()**self.q) / self.q ?

mvandenhi commented 2 years ago

Hey Sunnie, in your formulation you mustn't use detach(), as you still need to take the derivative there. While in their formula, the authors notice that the derivative of the GCE consists of the CE-derivative as well as the "loss_weight"(see p.5 of their paper). Hence, by fixing and detaching the "loss_weight" from the graph, for the backward pass they now only need to compute the gradient the cross_entropy function which is probably much faster. I don't understand why they multiply by the constant q though, do you? Best, Moritz