search

fancompute / neuroptica

Flexible simulation package for optical neural networks
https://doi.org/10.1109/JSTQE.2019.2930455
MIT License
208 stars 40 forks source link

loss function nan when using AbsSquared activation #3

Open twhughes opened 5 years ago

twhughes commented 5 years ago

running this mode

model_linear = neu.Sequential([
    neu.ClementsLayer(N),
    neu.Activation(neu.AbsSquared(N)),
    neu.DropMask(N, keep_ports=range(N_classes))
])

losses = neu.InSituAdam(model_linear, neu.CategoricalCrossEntropy, step_size=step_size).fit(x_train_flattened, y_train_onehot, epochs=n_epochs, batch_size=batch_size)

gives the warning:

  X_softmax = np.exp(X) / np.sum(np.exp(X), axis=0)
../neuroptica/neuroptica/losses.py:45: RuntimeWarning: invalid value encountered in true_divide
  X_softmax = np.exp(X) / np.sum(np.exp(X), axis=0)

And loss function is nan

When changing AbsSquared to Abs it works fine.

ianwilliamson commented 5 years ago

Noting from our discussion earlier: this appears to be from the polar form of the the derivative whenever r=0.