Is omega0 really needed ?

[grondilu]

Is it not possible to scale the input linear space instead ?

Also, not at all an issue, but I'd like to share the following Mathematica code attempting to replicate this, or at least the part for fitting an image :

image = ImageResize[ExampleData[{"TestImage", "Lena"}], 128]
{width, height } = ImageDimensions[image];
output = Flatten[ImageData[image] // #*2 - 1 &, 1];
linspace[n_] := Array[# &, n, {-#, #}] &[(n - 1)/2]
input = Pi/2*Tuples[{linspace[height], linspace[width]}] // N;
layer[n_, in_] := LinearLayer[
    n,
    "Weights" -> RandomReal[{-#, #}, {n, in}],
    "Biases" -> RandomReal[{-#, #}, {n}],
    "Input" -> in
] &[Sqrt[6/in]]
net = NetChain[
  {
   128, Sin,
   layer[128, 128], Sin,
   layer[128, 128], Sin,
   layer[128, 128], Sin,
   layer[3, 128], Sin
  },
  "Input" -> 2
]
net = NetTrain[net, (input -> output)]
Image[Partition[(# + 1)/2 &[net /@ input], width], ColorSpace -> "RGB"]
Table[NetExtract[net, {i, "Weights"}] // Flatten // Histogram, {i, {1, 3, 5, 7, 9}}]
Table[NetExtract[net, {i, "Biases"}] // Flatten // Histogram, {i, {1, 3, 5, 7, 9}}]

It works surprisingly well, but I haven't used omega0, I scaled the input instead. Also performance is better when the weights in the first layer are not initialized as advised in the paper. Not sure if it's related with the input scaling or what.

[vsitzmann]

Hi, indeed, scaling the input is equivalent to scaling omega_0 in the first layer - personally, it's easier for me to think about keeping the input normalized from (-1,1) and then think about omega_0 as the frequency of the signal, but that's a matter of preference! We actually got the initialization of the first layer wrong in the paper, the one in the implementation is correct - the first layer is very much signal-dependent in any case, so for instance, for audio, we set omega_0 to 3000 (at a (-1,1) input scale). Thanks for sharing the mathematica code, very cool!

[kwea123]

How did you determine the number, by trial and error?