Include L-BFGS-B optimizer

[sebasv]

The title is pretty self-explanatory. I think the setup of this project is great and I would like to add a great optimizer to it. I'll start to work on this if no one else has.

[to266]

Sure thing! Currently I don't have as much time to spend as I'd like, but BFGS would have been my next target, which would hopefully share a fair deal of code

[sebasv]

I have to say, after working with it for a couple of days, the idea of a single Minimizer trait starts to seem less appropriate. Different algorithms need different signatures to allow for differentiation between linear/nonlinear constraints, box constraints, scalar/vector minimization, etc. This also always bothered me about the scipy.optimize module. What are your thoughts about dropping the singular signature constraint?

[to266]

I was waiting for someone to play with the code a bit more to have some feedback regarding the interface. The goal was to kind of guess ahead what a somewhat reasonable interface might look like, build a single MVP for it, and then iteratively improving when more information is available, such as different minimizers that require different parameters, etc.

I think changing it is definitelly not off the table, however I expect the builder pattern to give a fair amount of flexibility. Could you post some examples where it's not very convenient or just straight-up impossible to use it?

As a side-note, feel free to ignore the Minimizer trait for now and just PR a working solution MVP. We can think what makes sense as a general interface then.

[to266]

@sebasv I thought about it some more about the Minimizer trait and I see your point. Do you think moving not-data to builders that are minimizer - specific make more sense?

Then all options will be stored and manipulated by the struct, and minimize would only take the data to be minimized on.

[sebasv]

As of right now, that is what I would prefer. I'm implementing some algorithms in that direction, see how it works.

[sebasv]

I ended up working mostly with Nelder-Mead because my problem seems to be too ill-posed for L-BFGS-B, but made some major changes in the project structure and the algorithm. The changes in the latter yield faster iterations and allows for convex constraints. I think I deviated too much from your codebase to make a sensible pull request, but maybe you can check out my code and we can cherry-pick changes to PR separately? code

[to266]

I doubt it's too different for a PR, but I'll have a look, thanks. None the less, great work! I'll definitely try to include any optimisations