search

fancompute / neuroptica

Flexible simulation package for optical neural networks
https://doi.org/10.1109/JSTQE.2019.2930455
MIT License
208 stars 40 forks source link

Sketchy Loss Functions #4

Closed twhughes closed 5 years ago

twhughes commented 5 years ago

When I train simple linear models, the loss function oscillates wildly. For example, using inSituAdam:

image 
model_linear = neu.Sequential([
    neu.ClementsLayer(N),
    neu.Activation(neu.Abs(N)),
    neu.DropMask(N, keep_ports=range(N_classes))
])

losses = neu.InSituAdam(model_linear, neu.CategoricalCrossEntropy, step_size=step_size).fit(x_train_flattened, y_train_onehot, epochs=n_epochs, batch_size=batch_size)

This may be a sign that the gradients are incorrect. Should double check.

twhughes commented 5 years ago

I think its fine. This is probably due to the way the batching works.

sunilkpai commented 5 years ago

Just got this email notif, I don't think you would get this issue if you removed the abs nonlinearity and did a meansquareerror loss.