Closed JDvorak closed 7 years ago
Hi @JDvorak, Thanks for trying out our code.
Could you point me to the tf version you are using? I have not met this problem before, 3 suggestions here:
try out tf 1.1, this is the version we test our code on. Might be issues in different versions.
replace self._grad_squared.append(tf.square(g) ) with maybe self._grad_squared.append(tf.square(tf.convert_to_tensor(g) ) )
You can also give me a minimal example here to reproduce.
Haven't tried it with an var_list
argument, but other than that, it worked fine for me with both tf 1.1. and 1.2 in general using cost function involving basic Tf math ops (e.g,. square). Maybe check the type of vae_discriminator_loss
and q_vars
?
I think #6 fixes this -- it simply checks to see if gradients are None. I think that TensorFlow can return "None" as a gradient in some cases, though I am unsure of exactly when this can occur.
Gradient check for "None" added in commits here https://github.com/JianGoForIt/YellowFin/commit/0949dfe2e674e1b83bdd06bbc21cf9b2d920ae1f from @jmhessel PR. Thanks @jmhessel
If I switch back to Adam, it works fine. Not sure what is up.