global layer normalization

[fjiang9]

Hi, Efthymios Thanks for sharing the code! Is this code https://github.com/etzinis/sudo_rm_rf/blob/5bf8c48d091c6adc74f834098692dc0386d961f0/sudo_rm_rf/dnn/models/sudormrf.py#L116 the "global layer normalization" mentioned in the paper?

[etzinis]

No this is layernorm.

Check the improved version where there is also the global norm layer https://github.com/etzinis/sudo_rm_rf/blob/5bf8c48d091c6adc74f834098692dc0386d961f0/sudo_rm_rf/dnn/models/improved_sudormrf.py

On Tue, Jul 13, 2021 at 7:38 AM Fei Jiang @.***> wrote:

Hi, Efthymios Thanks for sharing the code! Is this code

https://github.com/etzinis/sudo_rm_rf/blob/5bf8c48d091c6adc74f834098692dc0386d961f0/sudo_rm_rf/dnn/models/sudormrf.py#L116 the "global layer normalization" mentioned in the paper?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/etzinis/sudo_rm_rf/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFOOZCVBJGOOXLKO6UHED63TXQXVVANCNFSM5AJAUDQQ .

--

---

Efthymios Tzinis University of Illinois at Urbana Champaign (UIUC) PhD Candidate in Computer Science Siebel Office: 3332 | 201 N. Goodwin Ave. Urbana, IL, 61801, USA

[fjiang9]

I think this implementation makes no difference with nn.GroupNorm(1, nOut, eps=1e-08). Here is the test code: import torch import torch.nn as nn

x = torch.rand(2, 4, 3) ln = nn.GroupNorm(1, 4, eps=1e-8) print(ln(x)) print((x-x.mean(dim=[1, 2], keepdims=True))/(x.var(dim=[1, 2], keepdims=True, unbiased=False)+1e-8).sqrt()) # This is equivalent to your implementation except for the affine operation