karpathy / convnetjs

Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
MIT License
10.8k stars 2.04k forks source link

Is there a missing sqrt in the adadelta calculations? #78

Closed SundeepK closed 7 years ago

SundeepK commented 7 years ago

I was wondering if there is a missing sqrt call on this line https://github.com/karpathy/convnetjs/blob/master/src/convnet_trainers.js#L117

According to this article (links to adadelta paper)http://sebastianruder.com/optimizing-gradient-descent/index.html#adadelta it would suggest that:

var dx = - Math.sqrt((xsumi[j] + this.eps)/(gsumi[j] + this.eps)) * gij;

should be:

var dx = - (Math.sqrt(xsumi[j] + this.eps)/Math.sqrt(gsumi[j] + this.eps)) * gij;

Would be great if someone could correct my understanding if not.

SundeepK commented 7 years ago

Ah looks like I miss read the parenthesis! Looks fine.