search

karpathy / convnetjs

Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
MIT License
10.8k stars 2.04k forks source link

Fixed bias correction error in adam optimizer #107

Closed Chris-Nicholls closed 5 years ago

Chris-Nicholls commented 5 years ago

The algorithm described in section 2 of the Adam paper (https://arxiv.org/pdf/1412.6980.pdf) differs slightly from the current implementation.

image

The lines for m-hat and v-hat correspond to the lines in this pull request for biasCorr1 and biasCorr2.

ratajczak commented 5 years ago

it duplicates #70