Notes about BNN implementation

[stevengogogo]

What is Bayesian neural network?

• The bayesian neural network is a standard neural network with weights in a probability distribution.

What is the difference in training?

• The learning process depends on Bayes approach: $P(\theta | D) = \frac{P(D|\theta) P(\theta)}{P(D)} \propto P(D|\theta) P(\theta)$
• The trick to avoid $P(D)$ is by using sampling methods.
• Sampling methods includes
  • Metropolis-hasting Monte Carlo
  • Hamitonian Monte Carlo
  • Varitional Inference
  • Stochastic Langevin Descent

Implementation

For #5, I would recommend using Jax-Bayes that provides an example of BNN with regression problem. The bnn code for regression has been tested on my laptop, it works as expected.

The idea is to apply this method to #5, since there dataset is organized, and can be easily split into train-test set. We could use Jax framework to train the model with mcmc and variational inference

Computational environment

• Since JAX can be easily installed on Colab TPU, we could transfer to Colab for massive computation

Materias

• I highly recommend this article -- "Hands-on Bayesian neural network" that provides everything we need to know when implementing a BNN
  • This article provides source code to implement BNN with Pytorch.
  • This tutorial provides classification example
• The Bayesian method series (hierarchical linear model, hierarchical BNN, BNN) are another good resources on the statistical side.
  • This tutorials provides classification example
  • The source code of hierarchical BNN based on PyMC with Theano backend.
• JAX and BNN

[stevengogogo]

Introduction to Bayesian interpretation of weight decay

https://www.youtube.com/watch?v=vEPQNwxd1Y4