In your code it seems like you are trying to create a multilayer encoder, however what is actually happening is that multiple single layer decoders are being created. As you can see in this graph image created by tensorflow http://imgur.com/a/jqAn5 the rnn inputs are feed into each bidirectional rnn rather than one feeding into the other. I created this image by running your encoding_layer function and passing in some placeholders and then using the tf.summary.FileWriter to draw the graph.
To create a multilayer encoder you would want something like this
def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):
'''Create the encoding layer'''
layer_input = rnn_inputs
for layer in range(num_layers):
with tf.variable_scope('encoder_{}'.format(layer)):
cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,
initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw,
input_keep_prob = keep_prob)
cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,
initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw,
input_keep_prob = keep_prob)
enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw,
cell_bw,
layer_input,
sequence_length,
dtype=tf.float32)
layer_input = tf.concat(enc_output, 2)
# Join outputs since we are using a bidirectional RNN
enc_output = tf.concat(enc_output,2)
return enc_output, enc_state
You can see in this graph image http://imgur.com/a/bJANa That this creates the multi layer structure I assume you are going for. (The caveat is that in this function your rnn size must be half of the embedding size so if you want them to be different just move the first birnn out of the loop and have the layers after the first have the same size)
In your code it seems like you are trying to create a multilayer encoder, however what is actually happening is that multiple single layer decoders are being created. As you can see in this graph image created by tensorflow http://imgur.com/a/jqAn5 the rnn inputs are feed into each bidirectional rnn rather than one feeding into the other. I created this image by running your
encoding_layer
function and passing in some placeholders and then using thetf.summary.FileWriter
to draw the graph.To create a multilayer encoder you would want something like this
You can see in this graph image http://imgur.com/a/bJANa That this creates the multi layer structure I assume you are going for. (The caveat is that in this function your rnn size must be half of the embedding size so if you want them to be different just move the first birnn out of the loop and have the layers after the first have the same size)