Some questions about autoencoder！

[Imorton-zd]

Hi, I hope I'm not bothering you. Recently, I have implemented a simple autoencoder with keras for text classification to do domain adaptation, but it performs worse than the original representation of the document, about 10% lower. As the paper shown, stacted denoising autoencoder could improve the performance for domain adaptation. Would you help me check the errors and give me some suggestions? Thanks!

def autoencode(data):
    n_obs = data.shape[0]
    input_size = data.shape[1]
    encoding_size = 1000

    x = Input(shape=(input_size,))
    intermed = Dense(encoding_size * 2, activation='relu')(x)
    intermed = Dense(encoding_size * 2, activation='relu')(intermed)
    z = Dense(encoding_size, activation='relu', name='z')(intermed)
    intermed = Dense(encoding_size * 2, activation='relu')(z)
    intermed = Dense(encoding_size * 2, activation='relu')(intermed)
    x_reconstruction = Dense(input_size, activation='relu', name='x_reconstruction')(intermed)

    train_model = Model(input=[x], output=[x_reconstruction])
    test_model = Model(input=[x], output=[z])
    train_model.compile(optimizer='Adam',
                  loss = 'mse')
    test_model.compile(optimizer='Adam',
                  loss = 'mse')

    train_model.fit(x = data,
              y = data,
              nb_epoch=8)    

    encoded1 = train_model.predict(data)
    encoded2 = test_model.predict(data)
    return encoded1,encoded2

[EderSantana]

does the paper recomends training the entire model all at once? did you try layer wise pre-training?

[Imorton-zd]

@EderSantana The paper doesn't mention if the entire model is trained at once. Would you explain how to do layer wise pre-training simply? I am sorry that I have never tried it. Thanks!

[EderSantana]

it is called greedy-layer wise pretraining. its a little bit of a pain to do, but you can find tutorials online. essentially you trian an autoencoder for each layer at time

better than that, I just noticed that keras has an implementation of variational autoencoders in the examples folder. I think you should try that first. that one you don't need layer wise pretraining as much https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py