VAE

[ehfo0]

on VAE : why did u use 2 layer of encoder and 3 layers of decoder?

 def encoder(x):

    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']),
                                   biases['encoder_b1']))
    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']),
                                   biases['encoder_b2']))
    return layer_2

# Building the variational
def variation(x):
    mu = tf.add(tf.matmul(x, weights['hidden_mu']), biases['hidden_mu'])
    sig = tf.add(tf.matmul(x, weights['hidden_sig']), biases['hidden_sig'])

    layer_mu = tf.nn.sigmoid(mu)
    layer_sig = tf.nn.sigmoid(sig)
    epsilon = tf.random_normal(tf.shape(layer_sig), stddev=0.001, name='epsilon')
    layer_latent = tf.add(layer_mu, tf.multiply(tf.exp(0.5*layer_sig), epsilon))

    KLD = -0.5 * tf.reduce_sum(1+sig-tf.pow(mu, 2)-tf.exp(sig), reduction_indices=1)

    return layer_latent, KLD, epsilon, layer_mu

# Building the decoder
def decoder(x):
    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']),
                                   biases['decoder_b1']))
    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']),
                                   biases['decoder_b2']))
    layer_3 = tf.nn.relu(tf.add(tf.matmul(layer_2, weights['decoder_h3']),
                                   biases['decoder_b3']))
    return layer_3

shouldn't they be equal? and how can we get a better result?

[musicmilif]

Hi @ehfo0 : Thanks for watching and give some feedback. Yes, your are right, the number of layers should be equal in encoder and decoder. The reason why I use differen't number is my computer have only two core i5 CPU, so I replace last layer of encoder with variational layer to speed up my process.

I don't know a good way to modify the model, I just try several different activate function and number of layer, see the output picture and loss funtion to varify the model. You can see many script use relu for all activate function on MNIST. Although I haven't finish this code, in this data set I found sigmoid performs better.