Closed pieterblok closed 3 years ago
Hello @pieterbl86 , and sorry for the late reply, work and stuff was being very pushy. Anyway, thank you so much for suing BLiTZ.
In answer to your main question, there is actually no answer. The literature states that there is not yet a certain decision rule, but there are some ideas on how to follow.
When I use BLiTZ, I check the calibration of the model by seeing, for different Xs, what is the accuracy (or precision, or f1-score ou whatever metric I try) that the model give if I only keep the prediction of the X% less disperse predictions, getting the distribution of each prediction probability via monte-carlo sampling.
For the last layer, using monte-carlo sampling + softmax on top is the way I see on literature, on TensorFlow Probability and on research with my friends and peers.
Hope that helps. Feel free to reach me on my email or here if you have any doubts (I promise I will answer faster).
Best regards, -Pi
Colsing due to staleness.
Hello @piEsposito , thank you very much for this nice pytonic implementation of Bayesian neural nets!
Excuse me for this massive posting, it contains my inference script (which is a bit lengthy).
I have used your training script example (blitz/examples/bayesian_LeNet_mnist.py) to train a Bayesian CNN on the MNIST dataset. Then, I made an inference script using the training weights:
My main question: what is a good "decision rule" to select the MNIST-digits the BayesianCNN is less confident about?
As you can see in my script (pred_var == 0), I sampled the BayesianCNN 100 times and then rejected a digit when the variance of the 100 estimates exceeded 0 (meaning that the prediction is rejected when at least one of estimates deviates from the rest). I have also done a sanity check by simulating 64 random noise images, and then checking whether the BayesianCNN is giving uniform predictions there...
This is one of the outputs that was generated:
Logically, these 5 predictions (all predictions were of digit '8' by the way) are much better than the 64 predictions that were made with a standard LeNet trained on MNIST (without the Bayesian Layers).
Another question: the prediction of the BayesianCNN was done with the torch.nn.functional.softmax function (followed by the selection of the highest probability of the softmax). However I'm wondering if this softmax approach is correct... what would you advise? Is there a better probabilistic/Bayesian way?
Thanks in advance!