Adding variable to summaries changes training results

[olegarch]

I've added prob value from dropout as scalar summary to confirm that it is trainable and its value is changing from step to step. prob = tf.get_variable("prob", [], initializer=tf.constant_initializer(prob), trainable=True) tf.scalar_summary(prob.op.name, prob) Just adding it to summaries changed the results of the training. Is it expected?

[ilblackdragon]

That is quite strange, a summary is just a way to pull out a value when you are running the graph. It is not generating backpropagation or anything to update the variables.

Let me run an experiment to see what's going on.

[ilblackdragon]

Apparently adding summary changes something in random number generator order and produces different initalization values for variables. This explains why when you add a scalar - it results in slihtly different results, you may want to try to train for longer in your case to stabilize. (the bug with dropout probability fixed, thanks for reporting!).

Here is minimal code reproduce (I'll post it to TF main thread):

import tensorflow as tf

with tf.Session('') as sess:
    tf.set_random_seed(42)
    x = tf.placeholder(tf.float32, [1, 1])
    #tf.scalar_summary('d', 1.)
    y = tf.get_variable('y', [1, 5])
    d = tf.matmul(x, y)
    tf.merge_all_summaries()
    sess.run(tf.initialize_all_variables())
    print sess.run(d, feed_dict={x: [[1.]]})

Commenting / uncommenting scalar_summary changes results: [[ 1.25666416 0.54304159 1.69930065 1.39638388 0.80949366]] to [[-1.7023375 1.63059771 -1.05339909 -1.40406322 -1.13437796]]

[ilblackdragon]

Seems that this WAI, so I'm closing this issue.