search

kumar-shridhar / PyTorch-BayesianCNN

Bayesian Convolutional Neural Network with Variational Inference based on Bayes by Backprop in PyTorch.
MIT License
1.43k stars 325 forks source link

about resume or testonly #31

Closed ShellingFord221 closed 4 years ago

ShellingFord221 commented 5 years ago

Hi, I save the best model of BCNN in training and then resume from the checkpoint, I found that the accuracy of training has dropped from 90%+ to 80%+. I also complete the testOnly option, but I found that test accuracy of the saved best model is so much poorer then the best test accuracy when doing training and test in turns. I guess that since the weight is a distribution, the performance of best saved model can still fluctuate in a certain range. But if so, I can hardly save the best parameters and directly use it in the next time. So what do you think of this? Thanks!

kumar-shridhar commented 5 years ago

Hi @ShellingFord221 ,

The accuracy fluctuates a bit because of sampling from the distribution. The idea is to sample many times and then average over it to make sure it a bit more consistent. In the experiments, the best results were for samples between 10-30. But sometimes 100 were giving good results but they were heavy on memory. So it is a tradeoff.

Let me know it this helps.

ShellingFord221 commented 5 years ago

Hi, I have tried to sample 10, 30, 50, 100 times with testOnly option, all the results get very poor accuracy, although 50 times is the best among them. It seems like no matter how many times you sample when testing the best saved model, the model just overfits the test data. But it is saved because of the best test accuracy! Do you have this problem when testing only? I really wonder why.

kumar-shridhar commented 5 years ago

What task are you working on? On CIFAR10 the results during test times were not very different.