search

heykeetae / Self-Attention-GAN

Pytorch implementation of Self-Attention Generative Adversarial Networks (SAGAN)
2.53k stars 475 forks source link

Does it make sense to do spectral normalization before batch normalization? #9

Open richardwth opened 6 years ago

richardwth commented 6 years ago

Would the spectral norm get canceled out because it appears on both the nominator and denominator of the batch normalization equation?

I mean: bn(x*w/sn(w)) = gamma * (x*w/sn(w) - mean(x*w/sn(w))) / std(x*w/sn(w)) + beta = bn(x*w)

krishnatoshniwal commented 5 years ago

@richardwth Did you end up experimenting with any other order?

lzw-all-in commented 5 years ago

but i think std(xw/sn(w)) = std(xw) / (sn(w)^2)

richardwth commented 5 years ago

but i think std(xw/sn(w)) = std(xw) / (sn(w)^2)

No... std(x c) = std(x) c.

jerofad commented 5 years ago

And why was the 2 used even at the same time. I dont seem to understand why Spectral Norm and BatchNorm are used together. Can someone explain?

mcstarioni commented 5 years ago

They are orthogonal by their effect. SN changes weights of the layer, BN changes activations.

15732031137 commented 4 years ago

Hello! Thank you for your contribution to generative adversarial network research and for sharing your code! I am from China. Now is the Chinese New Year. I wish you a happy Chinese New Year! I am very interested in your thesis, when I try to add spectral normalization in my new networks, the program gives the following error: Traceback (most recent call last): File "SR.py", line 45, in train(opt, Gs, Zs, reals, NoiseAmp) File "E:\SinGAN-masterplus\SinGAN\training.py", line 34, in train D_curr,G_curr = init_models(opt) File "E:\SinGAN-masterplus\SinGAN\training.py", line 310, in init_models netG.apply(models.weights_init) File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 293, in apply module.apply(fn) File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 293, in apply module.apply(fn) File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 294, in apply fn(self) File "E:\SinGAN-masterplus\SinGAN\models.py", line 215, in weightsinit m.weight.data.normal(0.0, 0.02) File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 591, in getattr type(self).name, name)) AttributeError: 'Conv2d' object has no attribute 'weight'

I have searched a lot of information and couldn't solve it, so I want to ask you, I wish you a happy life, and look forward to your reply!