search

JavierAntoran / Bayesian-Neural-Networks

Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace, SG-HMC and more
MIT License
1.83k stars 302 forks source link

[Question][MC dropout] drop hidden units for each data point or for each mini-batch data? #20

Open alwaysuu opened 3 years ago

alwaysuu commented 3 years ago

Hi JavierAntoran,

Thanks for the wonderful code first, and it is really helpful for me working in the related area. I'd like to consult a question about the MC dropout. In BBB with local reparameterization, the activation values are sampled for each data point instead of directly sampling a weight distribution to reduce the computational complexity. So, in MC dropout, shall we do the similar procedure, e.g. dropout hidden units for each data point in training or testing phase? I notice that your MC dropout model seems uses the same dropout for a mini-batch data and the default batch size is 128. Should I change the batch size to 1 to achieve the goal of dropping hidden units for each data point?

Looking forward to your reply. Thanks a lot

JavierAntoran commented 2 years ago

Hi @alwaysuu,

Yes, changing the batch size to 1 will result in different weights being used for each input. However, it could make training very slow due to large variance in the estimate of the loss.

Javier