A question on the context pathway

[pykao]

Hello,

In the code network_architecture, from line 69 to 72, you did not apply BatchNormalization for skip1 layer.

l = ElemwiseSumLayer((l, r))
skip1 = NonlinearityLayer(l, nonlin)
if do_norm:
    l = BatchNormLayer(l, axes=axes)
l = NonlinearityLayer(l, nonlin)

However, for skip2, skip3 and skip4, you did apply BatchNormalization for these layers.

l = ElemwiseSumLayer((l, r))
if do_norm:
    l = BatchNormLayer(l, axes=axes)
l = skip3 = NonlinearityLayer(l, nonlin)

Can you explain the reason why you did not apply the BatchNormalizatoin on the first skip layer?

Best, Po-Yu

[FabianIsensee]

Dear Po-Yu, I did not use BN for the entire first stage and that is why I also did not apply it to batch normalization. It was one of the many things I tested but honestly, it does not make a significant difference whether you do BN in the first stage or not. Regards, Fabian

[pykao]

Dear Fabian, Thank you for your explanation! Best, Po-Yu