a bug maybe with sine_init.

[kingstarcraft]

from modules.py at line62, the sine_init is

m.weight.uniform_(-np.sqrt(6 / num_input) / 30, np.sqrt(6 / num_input) / 30)

but I think is:

m.weight.uniform_(-np.sqrt(6 / num_input) * 30, np.sqrt(6 / num_input) * 30)

follow is my test code with 101 sin layers, the std of outputs is stability equals 0.7 when 30， and when I replace ` factorwith/ factor`, the std of output reduced to 0.

import torch
import numpy as np

dim = 1000
factor = 30
alpha = 1
x = torch.Tensor(np.zeros(dim, dtype=np.float32)).reshape([dim, 1])
w = torch.Tensor(np.zeros([dim,dim], dtype=np.float32))

inputs = torch.nn.init.normal_(x, 0, 1)
weights = torch.nn.init.uniform_(w, -alpha/dim, alpha/dim)

outputs = torch.sin(alpha*factor * weights@inputs)
print(torch.mean(outputs), torch.std(outputs))

weights = torch.nn.init.uniform_(w, -np.sqrt(6 / dim) * factor / alpha, np.sqrt(6 / dim) * factor / alpha)

for _ in range(100):
    outputs = torch.sin(alpha*weights@outputs)
    print(torch.mean(outputs), torch.std(outputs))

[kingstarcraft]

I misunderstood the paper