sudden drop in val_acc using Adam

[mittaldhruv]

epoch,acc,loss,val_acc,val_loss

0,0.893458329291,0.357344379275,0.495111731844,2.99819970397

1,0.975676921584,0.075197558605,0.16061452514,11.4251960882

2,0.982579580745,0.0528487163138,0.49563547486,4.24635753768

3,0.980025710682,0.0596494577685,0.469448324022,5.40370503801

4,0.98200941382,0.0565878458491,0.499825418994,4.33332586763

[mittaldhruv]

found the issue, will update later

[mittaldhruv]

that sudden drop was because of the epsilon value in Adam optimizer's update function

t <- t + 1 lr_t <- learning_rate sqrt(1 - beta2^t) / (1 - beta1^t) m_t <- beta1 m_{t-1} + (1 - beta1) g v_t <- beta2 v_{t-1} + (1 - beta2) g g variable <- variable - lr_t * m_t / (sqrt(v_t) + epsilon)

that epsilon value is 10^-8 by default, and therefore the negative part becomes very high when sqrt(v_t) is very low and epsilon is low, so to avoid that take epsilon value = [0.1, 0.01, 0.001]