search

plumerai / rethinking-bnn-optimization

Implementation for the paper "Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization"
https://arxiv.org/abs/1906.02107
Apache License 2.0
73 stars 16 forks source link

kernel_initializer="glorot_normal" #27

Open puhan123 opened 2 years ago

puhan123 commented 2 years ago

In birealnet.py, I observe that kernel_initializer is set as "glorot_normal" for Conv2d; However, in QuantConv2D, the kernel_initializer is set as "glorot_normal". Kernel_initializer ="glorot_normal" means the kernel weights are all set as 1 ?

Thank you very much.

lgeiger commented 2 years ago

Glorot normal draws samples from a truncated normal distribution centered on 0, so in this case it means binary weights will be randomly set to either -1 or 1.

puhan123 commented 2 years ago

Thank you very much. I still have one more doubt. Are both Conv2d and QuantConv2D all set to either -1 or 1?

lgeiger commented 2 years ago

Are both Conv2d and QuantConv2D all set to either -1 or 1?

Only weights of QuantConv2D layers will be binarized.

puhan123 commented 2 years ago

My problem is that I found that the weights of QuantConv2D obtained with initialization method "glorot_normal" is not +1 or -1. "glorot_normal" in Pytorch framework is as below: image

The example is as : image

The value of w is apparently not +1 or -1.
Thank you very much.