About your implementation

[HuangZiliAndy]

Hi, thanks for your wonderful code! I have a little problem about your implementation. When you normalize w martrix, I notice you used ww = w.renorm(2,1,1e-5).mul(1e5). I think a natural implementation is just ww = F.normalize(w, dim=0). Do you have some special reasons for that? Thank you very much!!!

[PkuRainBow]

I also have similar concern, and I try to use the below function

nn.utils.weight_norm()

I read the doc, it says that,

Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by name (e.g. "weight") with two parameters: one specifying the magnitude (e.g. "weight_g") and one specifying the direction (e.g. "weight_v").

In fact, I want to normalize the weight vectors(L2 norm=1), so I am wondering whether the following function call can ensure that the weights' L2 norm equal 1 or it will learn a magnitude factor "weight_g"? So How can I ensure that the "weight_g" equal 1?

nn.utils.weight_norm(nn.Conv2d(512, num_classes, kernel_size=1, stride=1, padding=1, dilation=1, bias=False), name='weight')

[clcarwin]

There is no any special reason to use renorm. I just find it in pytorch's doc.

[douyh]

Hi, thanks for your wonderful code! I have a little problem about your implementation. When you normalize w martrix, I notice you used ww = w.renorm(2,1,1e-5).mul(1e5). I think a natural implementation is just ww = F.normalize(w, dim=0). Do you have some special reasons for that? Thank you very much!!!

Have you try F.normalize instead?