Open ywchan2005 opened 7 years ago
Hi @ywchan2005,
Thanks for trying out the optimizer. This is mostly because of the exploding gradient in the middle of training.
If it happens in the very beginning, you might want to play with the initial value a bit.
If it is in the middle of training, please consider using gradient clipping. There is discussion with solutions in our PyTorch YellowFin repo here. Similar solution can apply to the TF repo.
We are working on a better auto gradient clipping feature. You may also wait for that in a few days. But I suggest you can already start working on 2.
Thanks for the update. It happens mostly in the middle of training.
If I change the model to use relu (instead of elu), the issue doesn't happen anymore.
The same happened to me, first, you can a debug message to see exactly the values of the coefficients (where it gets the nan value) you can replace these lines in the get_mu_tensor(self)
method:
coef = tf.scatter_update(coef, tf.constant(2), -(3 + const_fact)) log_coef = tf.Print(input_=coef, data=[coef],message="coefficients: ") roots = tf.py_func(np.roots, [log_coef], Tout=tf.complex64, stateful=False)
Second, as @JianGoForIt suggested using clip_thresh parameter works for me.
However, another issue pops up, I get very high learning rate in the early stages in the training. That happens even in the case when my initial learning rate is very low (1e-5). The result is that the model diverges. Any ideas how to fix it?
Hi @IdanAzuri, Thanks for trying out our optimizer.
The estimators might be unstable when there are only a few iterations (as samples for the estimators in our optimizer). There are hacky ways to get around (clipping gradient / get a upper bound for lr in the begining iterations) but I would like to investigate whether there is systematical issues.
Could you please provide me a minimal example to reproduce your error?
@JianGoForIt, actually it's not stable after 400k iterations (until I stopped it) so it is not the problem. Sorry but I don't have a minimal example because my system is very complex with multiple subnets. clipping gradient / get an upper bound for learning rate doesn't work here, it still diverges.
I can confirm that I ran into this using a standard AlexNet architecture being trained on the ImageNet corpus using PyTorch. After 7 full epochs, (that is, having trained on 60928 minibatches, each of size 64) I received the following error:
yellowfin.py:192: RuntimeWarning: invalid value encountered in add
self._grad_var += global_state['grad_norm_squared_avg'] / debias_factor
/var/storage/shared/msrlabs/sabae/libsmolder_autodeploy/libsmolder/optimizers/yellowfin.py:329: RuntimeWarning: invalid value encountered in double_scalars
self._mu_t = max(root**2, ( (np.sqrt(dr) - 1) / (np.sqrt(dr) + 1) )**2 )
It would be really nice to have gradient clipping or some kind of workaround for this built-in to YellowFin. :)
Below is a simple piece of code to try YellowFin on my dataset.
Usually, it crashes and throws the following exception.