search

piEsposito / blitz-bayesian-deep-learning

A simple and extensible library to create Bayesian Neural Network layers on PyTorch.
GNU General Public License v3.0
933 stars 106 forks source link

module.log_prior is -inf #98

Open jordantkohn opened 2 years ago

jordantkohn commented 2 years ago

I'm implementing a BayesianLinear layer implemented like this: self.dense = BayesianLinear(opt.hidden_dim, opt.polarities_dim, freeze = False)

my loss from model.sample_elbo() returns "inf", and more specifically the module.log_prior() is "-inf". What could be causing this issue?

bayesian module: BayesianLinear( (weight_sampler): TrainableRandomDistribution() (bias_sampler): TrainableRandomDistribution() (weight_prior_dist): PriorWeightDistribution() (bias_prior_dist): PriorWeightDistribution() ) log_vp: tensor(-1425.3866, grad_fn=) log_prior: tensor(-inf, grad_fn=)'

piEsposito commented 2 years ago

Hi, please send me your snippet to reproduce the possible bug.

Em dom., 5 de dez. de 2021 às 18:31, Jordan Kohn @.***> escreveu:

I'm implementing a BayesianLinear layer implemented like this: self.dense = BayesianLinear(opt.hidden_dim, opt.polarities_dim, freeze = False)

my loss from model.sample_elbo() returns "inf", and more specifically the module.log_prior() is "-inf". What could be causing this issue?

bayesian module: BayesianLinear( (weight_sampler): TrainableRandomDistribution() (bias_sampler): TrainableRandomDistribution() (weight_prior_dist): PriorWeightDistribution() (bias_prior_dist): PriorWeightDistribution() ) log_vp: tensor(-1425.3866, grad_fn=) log_prior: tensor(-inf, grad_fn=)'

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/piEsposito/blitz-bayesian-deep-learning/issues/98, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLYRXR3DOEJNKTECWQZYODUPPKZTANCNFSM5JNHHTPA .

--

Pi Esposito | piesposito.github.io http://piesposito.github.io

jordantkohn commented 2 years ago

It's not really a self-contained example. But I've traced the -inf value back to this specific call in BayesianLinear module:

module.weight_prior_dist.log_prior(w)

jordantkohn commented 2 years ago

There are negative elements in w, which leads to 0 elements in prior_pdf. This causes -inf elements after taking the log of prior_pdf tensor.