Replicate paper results

[maxstrobel]

Hi,

thanks for this nice work, I really appreciate it! I tried to replicate the results from your paper with the repository, but I have not succeeded.

First, I downloaded your repo and the datasets. Then I adapted the configuration for the Bayesian Networks:

############### Configuration file for Bayesian ###############
n_epochs = 100
lr_start = 0.001
num_workers = 4
valid_size = 0.2
batch_size = 256
train_ens = 10
valid_ens = 10

Finally I run the evaluation script with main_bayesian.py --net_type alexnet --dataset CIFAR10, but the network is not able to overcome a validation accuracy of around 58%:

Epoch: 20   Training Loss: 2502238.8933     Training Accuracy: 0.5926   Validation Loss: 23935792.3000  Validation Accuracy: 0.5635     train_kl_div: 2445767.9315
Validation loss decreased (25103334.600000 --> 23935792.300000).  Saving model ...
Epoch: 21   Training Loss: 2387042.4522     Training Accuracy: 0.5968   Validation Loss: 22808838.9500  Validation Accuracy: 0.5584     train_kl_div: 2330986.6338
Validation loss decreased (23935792.300000 --> 22808838.950000).  Saving model ...
Epoch: 22   Training Loss: 2274617.4682     Training Accuracy: 0.6079   Validation Loss: 21713194.5000  Validation Accuracy: 0.5725     train_kl_div: 2219840.2803
Validation loss decreased (22808838.950000 --> 21713194.500000).  Saving model ...
Epoch: 23   Training Loss: 2166076.8439     Training Accuracy: 0.6137   Validation Loss: 20656232.4500  Validation Accuracy: 0.5872     train_kl_div: 2112406.7866
Validation loss decreased (21713194.500000 --> 20656232.450000).  Saving model ...
Epoch: 24   Training Loss: 2061628.5510     Training Accuracy: 0.6183   Validation Loss: 19644658.9000  Validation Accuracy: 0.5701     train_kl_div: 2008751.9745
Validation loss decreased (20656232.450000 --> 19644658.900000).  Saving model ...
Epoch: 25   Training Loss: 1961447.8232     Training Accuracy: 0.6230   Validation Loss: 18666660.6500  Validation Accuracy: 0.5711     train_kl_div: 1908918.2803
Validation loss decreased (19644658.900000 --> 18666660.650000).  Saving model ...
Epoch: 26   Training Loss: 1864639.9626     Training Accuracy: 0.6289   Validation Loss: 17726859.6500  Validation Accuracy: 0.5758     train_kl_div: 1812952.8240
Validation loss decreased (18666660.650000 --> 17726859.650000).  Saving model ...
Epoch: 27   Training Loss: 1771119.3846     Training Accuracy: 0.6386   Validation Loss: 16825135.8500  Validation Accuracy: 0.5862     train_kl_div: 1720880.1863
Validation loss decreased (17726859.650000 --> 16825135.850000).  Saving model ...
Epoch: 28   Training Loss: 1682560.0645     Training Accuracy: 0.6406   Validation Loss: 15963687.3750  Validation Accuracy: 0.5892     train_kl_div: 1632709.2596
Validation loss decreased (16825135.850000 --> 15963687.375000).  Saving model ...
Epoch: 29   Training Loss: 1597318.2373     Training Accuracy: 0.6459   Validation Loss: 15150667.4000  Validation Accuracy: 0.5615     train_kl_div: 1548427.4435
Validation loss decreased (15963687.375000 --> 15150667.400000).  Saving model ...
Epoch: 30   Training Loss: 1516623.9817     Training Accuracy: 0.6498   Validation Loss: 14361168.3500  Validation Accuracy: 0.5829     train_kl_div: 1467998.1879
Validation loss decreased (15150667.400000 --> 14361168.350000).  Saving model ...
Epoch: 31   Training Loss: 1439714.2970     Training Accuracy: 0.6520   Validation Loss: 13613963.9500  Validation Accuracy: 0.5829     train_kl_div: 1391386.5470
Validation loss decreased (14361168.350000 --> 13613963.950000).  Saving model ...
Epoch: 32   Training Loss: 1366105.2030     Training Accuracy: 0.6600   Validation Loss: 12909336.8000  Validation Accuracy: 0.5755     train_kl_div: 1318524.9443
Validation loss decreased (13613963.950000 --> 12909336.800000).  Saving model ...
Epoch: 33   Training Loss: 1296600.1863     Training Accuracy: 0.6617   Validation Loss: 12236651.6000  Validation Accuracy: 0.5815     train_kl_div: 1249338.7006
Validation loss decreased (12909336.800000 --> 12236651.600000).  Saving model ...
Epoch: 34   Training Loss: 1230397.9889     Training Accuracy: 0.6638   Validation Loss: 11600143.9500  Validation Accuracy: 0.5893     train_kl_div: 1183742.5000
Validation loss decreased (12236651.600000 --> 11600143.950000).  Saving model ...
Epoch: 35   Training Loss: 1168005.8073     Training Accuracy: 0.6705   Validation Loss: 11004782.7250  Validation Accuracy: 0.5683     train_kl_div: 1121634.4037
Validation loss decreased (11600143.950000 --> 11004782.725000).  Saving model ...
Epoch: 36   Training Loss: 1109223.4610     Training Accuracy: 0.6687   Validation Loss: 10435876.3750  Validation Accuracy: 0.5749     train_kl_div: 1062898.7377
Validation loss decreased (11004782.725000 --> 10435876.375000).  Saving model ...
Epoch: 37   Training Loss: 1053834.6206     Training Accuracy: 0.6691   Validation Loss: 9895180.6000   Validation Accuracy: 0.5803     train_kl_div: 1007417.1760
Validation loss decreased (10435876.375000 --> 9895180.600000).  Saving model ...
Epoch: 38   Training Loss: 1001452.7830     Training Accuracy: 0.6708   Validation Loss: 9391186.8750   Validation Accuracy: 0.5642     train_kl_div: 955054.9248
Validation loss decreased (9895180.600000 --> 9391186.875000).  Saving model ...
Epoch: 39   Training Loss: 951858.5939  Training Accuracy: 0.6717   Validation Loss: 8913133.8750   Validation Accuracy: 0.5767     train_kl_div: 905697.4590
Validation loss decreased (9391186.875000 --> 8913133.875000).  Saving model ...
Epoch: 40   Training Loss: 905384.9124  Training Accuracy: 0.6734   Validation Loss: 8459427.5000   Validation Accuracy: 0.5760     train_kl_div: 859194.2727
Validation loss decreased (8913133.875000 --> 8459427.500000).  Saving model ...
Epoch: 41   Training Loss: 861708.2651  Training Accuracy: 0.6720   Validation Loss: 8040532.7500   Validation Accuracy: 0.5767     train_kl_div: 815417.2926
Validation loss decreased (8459427.500000 --> 8040532.750000).  Saving model ...
Epoch: 42   Training Loss: 820970.3232  Training Accuracy: 0.6684   Validation Loss: 7639982.5250   Validation Accuracy: 0.5765     train_kl_div: 774222.8085
Validation loss decreased (8040532.750000 --> 7639982.525000).  Saving model ...
Epoch: 43   Training Loss: 782426.8826  Training Accuracy: 0.6698   Validation Loss: 7267536.7375   Validation Accuracy: 0.5746     train_kl_div: 735472.4554

Can you explain, how to replicate the results from the paper?

[maxstrobel]

push

[dhuruvapriyan]

I am also facing the same issue.

[kumar-shridhar]

Hi, Adding a beta term constraint over the KL divergence loss solves this issue.

Loss = NLL + KL

Adding a beta value multiplied with KL will solve the convergence issue.

Loss = NLL + ß * KL

where ß is a hyper-parameter.

We will update the repo soon with a good way to set ß.

[maxstrobel]

Do you have a rough estimate, when you will update the repo, e.g. days / weeks / months?

Thanks!

[kumar-shridhar]

In a week.

[ccpocker]

could you explain the parameter of train_ens and val_ens? Does it mean sample_number?

[kumar-shridhar]

Yes, that's correct.

[nikdn]

In a week.

Hello, with the latest version of the code still replicating the validation accuracy of over 80 percent on the cifar10 as stated in the paper (figure 3) is not successful. Would you be able to guide me a bit toward the configuration and initialization of the parameters?

[StevenLauHKHK]

@kumar-shridhar How to reproduce the validation accuracy as stated in the paper on the cifar10 dataset? The network cannot break through 64% in validation accuracy when I use the same setting in your configuration file. Thank you.