Closed cforssen closed 1 month ago
Based, e.g., on the blog post by Sebastian Ruder https://www.ruder.io/optimizing-gradient-descent/#gradientdescentoptimizationalgorithms and arXiv 1609.04747
Completed a first draft. There are no mathematical details for the momentum-based methods.
Replace the gradient descent chapter with a more general optimization one.
To be included:
[x] Mathematical definition of the optimization problem
[x] Local versus global minima
[x] Maybe (non-exhaustive) table with selected approaches
[x] Gradient descent (GD) general formulation
[x] GD: batch, stochastic, mini-batch versions including challenges
[x] Short on improvements incorporating momentum or momentum averaging