Implement Basic Optimization Suite

[cannontwo]

• [ ] SGD
• [ ] Brent's Algorithm in One Dimension
• [x] Generating Set Search (compass search/pattern search)
• [ ] CMA-ES
• [ ] Conjugate Gradient Descent
• [x] BFGS
• [x] Linear Programming
• [ ] Quadratic Programming
• [ ] SQP

[cannontwo]

May also want to define some canonical tasks in order to test/benchmark.

[cannontwo]

Nealder-Mead is empirically quick but may fail to find a stationary point on some problems. See here---specifically Algorithm 3.2---for information on Generating Set Search (compass search), which I'm implementing for derivative-free deterministic optimization instead. My implementation should use a minimal generating set (n+1 directions, defining a regular simplex in n dimensions) or coordinate search directions (2n sample directions). To accelerate search, periodically rotate the search directions by orienting a primary vector or a la Rosenbrock search (but use Palmer's innovation instead of Gram-Schmidt) respectively.

[cannontwo]

For the SQP solver, implement algorithm 18.3 from Nocedal & Wright. For the QP solver, implement algorithm 16.4, solving the linear system using the projected CG algorithm 16.2.

[cannontwo]

For Linear Programming, implement interior point method in Ch. 14 of Nocedal and Wright

[cannontwo]

For Linear Programming, implement interior point method in Ch. 14 of Nocedal and Wright

Tried this, but implementation is too complex to be worthwhile right now. For now, just using Gurobi.

[cannontwo]

Linear Programming implemented as wrapper around Gurobi in 7626ce8adaa5683000d7e89f54107d4f10dd8e63