search

torch / optim

A numeric optimization package for Torch.
Other
197 stars 154 forks source link

Question on rmsprop implementation #149

Closed syoungbaak closed 7 years ago

syoungbaak commented 7 years ago

According to Tensorflow implementation, it seems to me that the 53rd line of 'rmsprop.lua' needs to modified as

state.tmp:sqrt(state.m):add(epsilon) --> state.tmp:sqrt(state.m+epsilon)

Is it okay to use the original one without modification when I try to train Inception-resnet v2 from the scratch uisng the same optimization parameters ?

alirezag commented 7 years ago

This is just a dampening factor, you can get approximately similar behavior if you use epsilon = sqrt(tensorflow_epsilon)