Adding layer

[amobiny]

Hey, As you might remember, I opened an issue on your other capsnet repo. and asked about adding layers. I used this repo (Capslayer) and saw the same thing. Say just adding one fully connected layer (similar to digitcap but with different number of capsule, say 20). This is the only part of code that I'm changing:

return digitCaps: [batch_size, 20, 16, 1], activation: [batch_size, num_label]

with tf.variable_scope('DigitCaps_layer1'):
       primaryCaps = tf.reshape(primaryCaps, shape=[cfg.batch_size, -1, 8, 1])
       digitCaps1, activation = capslayer.layers.fully_connected(primaryCaps, prim_act, num_outputs=20, 
 out_caps_shape=[16, 1], routing_method='DynamicRouting')

return digitCaps: [batch_size, num_label, 16, 1], activation: [batch_size, num_label]

with tf.variable_scope('DigitCaps_layer2'):
        self.digitCaps, self.activation = capslayer.layers.fully_connected(digitCaps1, activation, 
 num_outputs=10, out_caps_shape=[16, 1], routing_method='DynamicRouting')

The training and validation accuracy goes down to around 10%! I thought maybe it's because the number of parameters which increases from 8 million to 10 million by adding this single layer