Add more Optimization / Minimization algorithms

[Recktenwald]

Is your feature request related to a problem? Please describe. Currently, there is no way to find the argument / function value that minimizes a target function. Such operations are often useful.

Describe the solution you'd like I suggest implementing a relatively small set of algorithms that can handle a wide range of target functions. Specifically, I suggest implementing

• The Nelder-Mead Method, as it does not require any special properties of the target function or any derivative-information. Also, it is relatively simple.
• A very general solver for equality and inequality constrained problems. In particular, I suggest using Sequential Quadratic Programming.

This Sequential Quadratic Approach is also used in the default method chosen by the scipy.optimize.minimize function. The scipy package is however relying on a Fortran implementation, and I'm not sure if F# native performance will be enough. There is a big suite of Test functions for optimization, and benchmarking could be done with a handful of these.

Describe alternatives you've considered Use MathNET.Numerics instead, and write some glue code. Also, it could be discussed if a different set of algorithms would make more sense. I'm not sure about the performance tradeoffs of using SQP as the de-facto standard for everything, that cannot be managed with Nelder-Mead.

Additional context Add any other context or screenshots about the feature request here.

[bvenn]

In the past we already had some optimizations, that we had to omit because of dependencies of the Microsoft solver foundation (see here). It would be great to have these optimizations back in FSharp.Stats :+1:

[bvenn]

add nelder mead c3a98d412c2f58d992d5ea0b46995b9092bae86e