search

IntelLabs / bayesian-torch

A library for Bayesian neural network layers and uncertainty estimation in Deep Learning extending the core of PyTorch
BSD 3-Clause "New" or "Revised" License
483 stars 67 forks source link

Batchnorm layer #16

Closed fzimmermann89 closed 1 year ago

fzimmermann89 commented 1 year ago

Hi,

i am rather new to the concept of BNNs and wondering if there is a reason for the BatchNorm affine transformation parameters not being considered as random variables, i.e. why the transformation is deterministic?

Furthermore, batchnorm always returns the KL norm (as zero ) whereas for other layers one can disable it for better integrarion with pytroch sequential wrapper etc..

Overall, I am confused: How is the current batchnorm implementation different from vanilla pytroch except returning a constant zero as second value?

Thans in advance Felix

ranganathkrishnan commented 1 year ago

Hi @fzimmermann89, Right now the library does not support Bayesian form of BatchNorm layers, however the Bayesian layer for BatchNorm with parameters as random variables can be implemented similar to Linear/Conv layers (pull requests are welcome).

The BatchNorm layer is just a wrapper around vanilla PyTorch to return a tuple (constant zero as second value), so that it binds with other Bayesian layers within Sequential model definition. However you may not need to use this wrapper if you are building the model using dnn_to_bnn() API {example}.