search

kumar-shridhar / PyTorch-BayesianCNN

Bayesian Convolutional Neural Network with Variational Inference based on Bayes by Backprop in PyTorch.
MIT License
1.42k stars 323 forks source link

discrepency in you AlexNet model #25

Closed Coderx7 closed 4 years ago

Coderx7 commented 5 years ago

Hello. I noticed you didnt include the kl term for the CNN layers in your loss. was this intentional since the previous AlexNet model, still uses that term!

kumar-shridhar commented 5 years ago

Hi @Coderx7 ,

Can you please be a bit more specific as this term in Conv layer: image includes the KL in the overall loss.

Coderx7 commented 5 years ago

Hi, in Image Recognition/utils/BayesianModels/BayesianAlexNet.py, if you look closely you can see that, you are actually only taking into account the kl regarding the fc layers (i.e. the classifier). 

def probforward(self, x):
    kl = 0
    for layer in self.layers:
        if hasattr(layer, 'convprobforward') and callable(layer.convprobforward):
            x, _kl, = layer.convprobforward(x)
        else:
            x = layer.forward(x)
    x = x.view(x.size(0), -1)
    x, _kl = self.classifier.fcprobforward(x)
    kl += _kl
    logits = x
    return logits, kl

the line 

x, _kl = self.classifier.fcprobforward(x)

overwrites the previous kl from the convlayer . this however is not the case in `BayesianAlexNet.py` and in that file, the kl is accounted for, for both types of layers.

JurijsNazarovs commented 5 years ago

Hello @Coderx7, did you try to fix it with kl += _kl as it was before? If yes, is result from the paper still reproducible?

Coderx7 commented 4 years ago

@JurijsNazarovs I didnt run the code, I was just eyeing through code and spotted that. by the way please accept my apologies for replying this late I just forgot about this!

suredie commented 4 years ago

Hello @Coderx7, did you try to fix it with kl += _kl as it was before? If yes, is result from the paper still reproducible?

No, by doing so, it seems the model cannot converge.

kumar-shridhar commented 4 years ago

Hi, Sorry I was busy with some other commitments. I have tried to fix this issue. Please check it out and let me know your thoughts.