How does the suggested initialization compare to the Xavier initialization method ?

[grondilu]

I've just found out about the so-called Xavier initialization method, which to me seems to do what the paper claims to be its main contribution.

It's an option to Mathematica's NetInitialize function.

image = ExampleData[{"TestImage", "Lena"}]

image = ImageResize[image, 128];

{width, height } = ImageDimensions[image]

output = Flatten[#*2 - 1 &@ImageData[image], 1]

linspace[n_] := Array[# &, n, (n - 1)/2 // {-#, #} &]

input = Pi/2 Tuples [{linspace[height], linspace[width]}] // N

net = NetInitialize[
  width // NetChain[
     {
      #, Sin,
      #, Sin,
      #, Sin,
      #, Sin,
      3, Sin
      },
     "Input" -> 2
     ] &,
  Method -> "Xavier"
  ]

net = NetTrain[net, (input -> output)];
Image[Partition[(# + 1)/2 &[net /@ input], width], ColorSpace -> "RGB"]

image

Do[
 Print[Table[
   NetExtract[net, {i, x}] // Flatten // 
    Histogram, {i, {1, 3, 5, 7, 9}}]],
 {x, {"Weights", "Biases"}}
 ]

[vsitzmann]

The proposed initialization method is derived independently under different assumptions, for a different nonlinearity, and with a different derivation than Xavier initialization. The assumptions made in Glorot et al. 2010 do not apply to the sine nonlinearity ("Consider the hypothesis that we are in a linear regime at the initialization..."). Remarkably, the final result turns out to be the same, but that was neither known, trivial, nor clear before our analysis - so indeed, this is a surprising finding of our paper!