Open Imperssonator opened 6 years ago
I've found in training on my own image dataset that I get an InvalidArgumentError: Nan in summary histogram. This was caused by the following line:
slopes = tf.sqrt(tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))
You'd think that the sum of squared values would never give anything less than 0, but apparently it does, because adding 1e-8 inside the sqrt() fixed the problem:
slopes = tf.sqrt( 1e-8 + tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))
I faced the same problem while training mnist with WGANGP, this solutions works fine for me. Thanks~
@Imperssonator faced the same problem too, that's a werid problem, thanks for saved my life!
I've found in training on my own image dataset that I get an InvalidArgumentError: Nan in summary histogram. This was caused by the following line:
slopes = tf.sqrt(tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))
You'd think that the sum of squared values would never give anything less than 0, but apparently it does, because adding 1e-8 inside the sqrt() fixed the problem:
slopes = tf.sqrt( 1e-8 + tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))