StillerPatrick
commented
2 years ago This bug I found has strong influence on the gradients. I computed it with the following example. This simulate the gradient of one finger:
model = torch.nn.Sequential(nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 1))
def f(x):
return model(x)
x = torch.ones(1, 3) * 2048
y = f(x)
y.backward()
print("not normalized", model[0].weight.grad)
lb = torch.tensor([0, 0, 0])
ub = torch.tensor([2048, 2048, 2048])
linear_layer = nn.Linear(3, 1)
model = torch.nn.Sequential(nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 3),
sin_activation(),
nn.Linear(3, 1))
def f(x):
x = 2 * (x - lb) / (ub - lb) - 1
return model(x)
x = torch.ones(1, 3) * 2048
y = f(x)
y.backward()
print("normalized", model[0].weight.grad)With the following output:
not normalized
tensor([[-25.9710, -25.9710, -25.9710],
[ 6.6366, 6.6366, 6.6366],
[125.2777, 125.2777, 125.2777]])
normalized
tensor([[ 0.0328, 0.0328, 0.0328],
[ 0.0176, 0.0176, 0.0176],
[-0.0114, -0.0114, -0.0114]])
This pull request includes a new version of the finger net that allows handling any spatial-temporal domain. Furthermore, there is a bug fix that ensures the correct normalization of the input data. The new version of the finger net is also tested by a unit test.