New chapter: Mathematical optimization

[cforssen]

Replace the gradient descent chapter with a more general optimization one.

To be included:

• [x] Mathematical definition of the optimization problem

  • Different terminology (minimization versus maximization)
  • Functionals and quantum variational problems as examples
  • Loss functions and data dependence

• [x] Local versus global minima

• [x] Maybe (non-exhaustive) table with selected approaches

• [x] Gradient descent (GD) general formulation

• [x] GD: batch, stochastic, mini-batch versions including challenges

• [x] Short on improvements incorporating momentum or momentum averaging

  • Adagrad
  • RMSprop
  • Adam [https://arxiv.org/abs/1412.6980]

[cforssen]

Based, e.g., on the blog post by Sebastian Ruder https://www.ruder.io/optimizing-gradient-descent/#gradientdescentoptimizationalgorithms and arXiv 1609.04747

[cforssen]

Completed a first draft. There are no mathematical details for the momentum-based methods.