Closed Spnetic-5 closed 1 year ago
CI seems to have trouble downloading test files from GitHub. Not real failures; tests pass on my end.
@Spnetic-5, just checking, your latest commit message mentions the convergence test but the commit doesn't actually include the test.
@Spnetic-5, just checking, your latest commit message mentions the convergence test but the commit doesn't actually include the test.
Sorry, forgot to push test file😅️
Thank you, @Spnetic-5, will merge. Please release v0.14.0 when ready.
Adam Implementation:
Momentum Update:
m = beta1 * m + (1 - beta1) * gradient
Adaptive Learning Rates Update:
v = beta2 * v + (1 - beta2) * gradient**2
Bias-Corrected First and Second Moment Estimates:
m_hat = m / (1 - beta1**t)
v_hat = v / (1 - beta2**t)
Parameter Update:
param = param - learning_rate * m_hat / (sqrt(v_hat) + epsilon)