Open quangnhien opened 4 years ago
I am also slightly confused regarding this implementation, In the bayes by backprop paper by blundell, the NLL loss is an expectation over the variational posterior weights. I am not sure why there is a multiplication by self.train_size
Nll_loss here is used to calculate the negative log likelihood - logP(D|wi) in the paper of blundell, In my opinion, here should not apply log_softmax ,calculate mean probs and then use mean probs and targets to get cross entropy(softmax+log+nll_loss). Instead, for each logits obtained from the sampling model, the cross-entropy should be directly calculated by logits and targets to obtain multiple negative log likelihood, and sum these losses according to the number of samples. This is the third term of the formula F(D,θ) in this paper. In addition, the name of the class should not be ELBO, beta*kl here corresponds to the -elbo loss in paper, but the kl here is actually calculate kl divergence between q and the prior, and the paper is to minimize -elbo, namely -elbo = logq (wi|θ) - logP (wi)
You can explain for me?.