search

JianGoForIt / YellowFin

auto-tuning momentum SGD optimizer
Apache License 2.0
422 stars 93 forks source link

Swap in replacement of AdamOptimizer causes crash #5

Closed JDvorak closed 7 years ago

JDvorak commented 7 years ago 
        self.opt_q =  YFOptimizer().minimize(self.vae_discriminator_loss, var_list=q_vars)

  File "xxx\src\yellowfin.py", line 215, in apply_gradients
    after_apply_op = self.after_apply()
  File "xxx\src\yellowfin.py", line 139, in after_apply
    self._grad_squared.append(tf.square(g) )
  File "C:\Miniconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 412, in square
    return gen_math_ops.square(x, name=name)
  File "C:\Miniconda3\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 2585, in square
    result = _op_def_lib.apply_op("Square", x=x, name=name)
  File "C:\Miniconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 509, in apply_op
    (input_name, err))
ValueError: Tried to convert 'x' to a tensor and failed. Error: None values not supported.
PS xxx>

If I switch back to Adam, it works fine. Not sure what is up.

JianGoForIt commented 7 years ago

Hi @JDvorak, Thanks for trying out our code.

Could you point me to the tf version you are using? I have not met this problem before, 3 suggestions here:

  1. try out tf 1.1, this is the version we test our code on. Might be issues in different versions.

  2. replace self._grad_squared.append(tf.square(g) ) with maybe self._grad_squared.append(tf.square(tf.convert_to_tensor(g) ) )

  3. You can also give me a minimal example here to reproduce.

rasbt commented 7 years ago

Haven't tried it with an var_list argument, but other than that, it worked fine for me with both tf 1.1. and 1.2 in general using cost function involving basic Tf math ops (e.g,. square). Maybe check the type of vae_discriminator_loss and q_vars?

jmhessel commented 7 years ago

I think #6 fixes this -- it simply checks to see if gradients are None. I think that TensorFlow can return "None" as a gradient in some cases, though I am unsure of exactly when this can occur.

JianGoForIt commented 7 years ago

Gradient check for "None" added in commits here https://github.com/JianGoForIt/YellowFin/commit/0949dfe2e674e1b83bdd06bbc21cf9b2d920ae1f from @jmhessel PR. Thanks @jmhessel