added learning rate bias correction and corrected m_t

openai / iaf

Code for reproducing key results in the paper "Improving Variational Inference with Inverse Autoregressive Flow"

https://arxiv.org/abs/1606.04934

MIT License

517 stars 130 forks source link

added learning rate bias correction and corrected m_t #15

Open dchatterjee172 opened 6 years ago

dchatterjee172 commented 6 years ago

Added learning rate bias correction and,
m_t = m.assign(tf.maximum(beta2_t * m + eps, tf.abs(grad)))
this line is wrong because if (beta2_t * m+eps)<0 and grad==0 then g_t = v_t / m_t will be undefined. correct line is,
m_t = m.assign(tf.maximum(beta2_t * m,tf.abs(grad)+eps))

dchatterjee172 commented 6 years ago

looks like correction for m_t was not needed. as (beta2_t * m+eps) is always >0