On sparse autoencoders

[superhans]

Hi Guys,

I was trying out a toy example using autoencoders in Keras. I have a input which has 10000 examples of size 174. I want to run the case where there is one sparse hidden layer is also of dimension 174. Is the code below correct ? No matter how hard I try, the training accuracy saturates at around 15%, which is too low. What should I be doing differently ?

autoencoder = Sequential()
encoder = containers.Sequential([Dense(174, 174, activity_regularizer = keras.regularizers.ActivityRegularizer(l1=0.))])
decoder = containers.Sequential([Dense(174, 174)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=True, tie_weights=True))

autoencoder.compile(loss='mean_squared_error', optimizer='rmsprop')  
loss = autoencoder.fit(data_shuffle,data_shuffle, nb_epoch=100, batch_size=50,show_accuracy=True, validation_split = 0.3);

The output is : Epoch 95 7014/7014 [==============================] - 1s - loss: 0.0004 - acc.: 0.1484 - val. loss: 0.0005 - val. acc.: 0.1407 Epoch 96 7014/7014 [==============================] - 1s - loss: 0.0004 - acc.: 0.1451 - val. loss: 0.0005 - val. acc.: 0.1347 Epoch 97 7014/7014 [==============================] - 1s - loss: 0.0004 - acc.: 0.1404 - val. loss: 0.0005 - val. acc.: 0.1460 Epoch 98 7014/7014 [==============================] - 1s - loss: 0.0004 - acc.: 0.1457 - val. loss: 0.0005 - val. acc.: 0.1414 Epoch 99 7014/7014 [==============================] - 1s - loss: 0.0004 - acc.: 0.1459 - val. loss: 0.0005 - val. acc.: 0.1357

[fchollet]

l1=0. in your ActivityRegularizer means your regularizer is doing nothing.

You seem to misunderstand the meaning of validation accuracy. For an input reconstruction problem it is meaningless, you should only refer to this metric in classification problems.

Here your input is near-perfectly reconstructed, most likely because the autoencoder has learned the identity matrix.