Closed victor-yon closed 2 years ago
I've had the same error, but with the BayesianGRU
instead.
The problem is caused by calculating log(0)
in the log_prior(w)
calculation of PriorWeightDistribution
in weight_sampler.py.
Because of log(0)=-inf
, the sampled weights partially get filled with nan
-values, which then causes the ValueError
when passed to pytorchs distribution library.
I fixed the problem for me like by changing PriorWeightDistribution.log_prior()
:
# OLD CODE
# prior_pdf can be zero due to nummerics --> -inf possible
return (torch.log(prior_pdf) - 0.5).sum()
# NEW CODE
# adding a tiny number (1e-6 in my case) resolves the problem
return (torch.log(prior_pdf+1e-6) - 0.5).sum()
@donhauser that's awesome. I always lose track of the points where we need to add those small values to keep numerical stability. If you want to PR I'll merge it.
I found a weird bug, I was able to reproduce it with the bayesian_LeNet_mnist example by reducing the number of parameters.
I have no idea why but based on my tests it occurs randomly during the training (though with the same seed it always trigger at the same iteration). And I was able to reproduce it only with small networks, if I add more neurons or more layers the problem never happen.
Error traceback
Code