search

itayhubara / BinaryNet.pytorch

Binarized Neural Network (BNN) for pytorch
505 stars 126 forks source link

why does the alexnet_binary has Hardtanh activation when alexnet has ReLU activation? #20

Open Ashokvardhan opened 5 years ago

Ashokvardhan commented 5 years ago

@itayhubara : I noticed that all the binarized neural network files alexnet_binary.py, resnet_binary.py, vgg_cifar10_binary.py have Hardtanh activation function whereas their respective parent architectures in the files alexnet.py, resnet.py, vgg_cifar_10 have ReLU activation function. Is there any specific reason for this? However the Theano implementaion of Binary Connect code here uses ReLU activation when we binarize just the weights.

itayhubara commented 4 years ago

I just demonstrated how to change the model to BNN model on ResNet and vgg_cifar. You should replace the ReLU with HardTanH in all other models as well. Basically you must ensure that you have BNN and the HardTanH before every binary GEMM operation.