Closed Leoshiou closed 5 years ago
Excuse me, can someone explain, why for batch normalization, it takes the maximum of the tensor of multiplying alpha and not as the following code.
if batch_norm: depth = shape[3] scale = tf.Variable(tf.ones([depth, ], dtype='float32'), name='scale') shift = tf.Variable(tf.zeros([depth, ], dtype='float32'), name='shift') mean = tf.Variable(tf.ones([depth, ], dtype='float32'), name='rolling_mean') variance = tf.Variable(tf.ones([depth, ], dtype='float32'), name='rolling_variance') conv_bn = tf.nn.batch_normalization(conv, mean, variance, shift, scale, 1e-05) conv = tf.add(conv_bn, biases) **conv = tf.maximum(self.alpha * conv, conv)** else: conv = tf.add(conv, biases)
I found out that it was the leaky activation function.
Excuse me, can someone explain, why for batch normalization, it takes the maximum of the tensor of multiplying alpha and not as the following code.