Error in NCE expression?

[bczhu]

In line 227 of nce_loss.py: logit_true = logit_model - logit_noise - math.log(self.noise_ratio), but this is not the same as the log of line 223:# p_true = logit_model.exp() / (logit_model.exp() + self.noise_ratio logit_noise.exp()) where is the logit_model in the denominator? shouldn't be logit_true = logit_model - log( logit_model.exp() + self.noise_ratio logit_noise.exp())?

[Stonesjtu]

1. There may be confusion on our termiology, logit_true != log(p_true), we feed this logit number to BCELoss.
2. The BinaryCrossEntropy loss in pytorch combines both sigmoid and cross entropy functions, it treats sigmoid(logit_true) as the probability however.

3. sigmoid(logit_true) = 1 / (1 +exp(-logit_true))
   = 1 / ( 1 + (logit_noise.exp() * self.noise_ratio) / logit_model.exp() )
   = logit_model.exp() / (logit_model.exp() + self.noise_ratio * logit_noise.exp())
   = p_true