about moving average

[hityzy]

I see that you add tf.contrib.layers.batch_norm in the networks, but I can't fine where do you update the moving average of bn

[duxingren14]

The self.d_optim and, self.g_optim will do it automatically. Please look at how d_vars and g_vars are defined. Thanks.

Zili

On 26 July 2017 at 22:39, hityzy notifications@github.com wrote:

I see that you add tf.contrib.layers.batch_norm in the networks, but I can't fine where do you update the moving average of bn

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/duxingren14/DualGAN/issues/3, or mute the thread https://github.com/notifications/unsubscribe-auth/ACc6e4A95GrRD0qwnNHebSzPm1hMUHi6ks5sSCJ4gaJpZM4Ok0ID .

[hityzy]

Thanks for your replay! Maybe I have made a mistake, however it is said in "https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm" that we should add update ops before optimizer like this: update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) with tf.control_dependencies(update_ops): train_op = optimizer.minimize(loss)

Maybe the vars for derivation may not contain moving average, because moving average is not updated by SGD.

[duxingren14]

I am sure moving average is a member of tf.trainable_variables.

What I understand about the document is, the SGD may update the moving average. However, that can have a speed penalty, especially in distributed settings. Maybe the UPDATE_OPS have been optimised for speed.

On 28 July 2017 at 00:46, hityzy notifications@github.com wrote:

Thanks for your replay! Maybe I have made a mistake, however it is said in "https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm" that we should add update ops before optimizer like this: update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) with tf.control_dependencies(update_ops): train_op = optimizer.minimize(loss)

Maybe the vars for derivation may not contain moving average, because moving average is not updated by SGD.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/duxingren14/DualGAN/issues/3#issuecomment-318585472, or mute the thread https://github.com/notifications/unsubscribe-auth/ACc6e2eG5ZzIRnMevgNOBYzSS69Gs1jdks5sSZHIgaJpZM4Ok0ID .