Prefer scipy.optimize.least_squares with 'trf' method over this leastsq extension?

[floschl]

Hi Jonathan,

With the release of SciPy version 0.17, SciPy comes with nonlinear least-squares solver which allows users to include bounds on the fit parameters, scipy.optimize.least_squares. The methods used in scipy.optimized.least_squares to inforce the bounds are much better than the method used in leastsqbound and the use of the SciPy method is encouraged.

This is very interesting. So you are recommending scipy.optimize.least_squares over the unconstrained LM+leastsqbound (so with your bounds extension), which seem also be used in higher-level optimisation frameworks (e.g. astropy, lmfit etc.).

I am wondering about which method in scipy.optimize.least_squares you exactly mean? The 'lm' method does not have bounds integrated, so do you mean the 'trf' method?

Thanks and all best, Florian

[newville]

@floschl you are right that the lm method in scipy.optimize.least_squares does not support bounds whereas the trf method does.

I am not certain that the Coleman-Li scaling used in trf is always better - I don't know that a careful comparison has been made or published. From my perspective, the transformation approach used here (first done in MINUIT, if I am not mistaken) has a very important feature that we exploit in LMFIT, namely that the bounds need not be a feature of the solver, but of the parameters, independent of the solver. So, while scipy.optimize.least_squares supports bounds when using trf but not lm method, lmfit supports bounds on parameters even when using the MINPACK implementation of lm. This allows users to change the solver method without have to change any bounds or any other aspect of their minimization problem.

[floschl]

@newville thanks for your answer. For my optimisation problems, I do have some issues with the constrained LM fitting that has been implemented in lmfit and also astropy (exactly the same results, I suppose the same kind of transformation is used in both frameworks). In contrast, the implementation of levmar (Python implementation) works better (a different constrained extensions seems to be implemented there with the aid of some LAPACK routines), but has some other drawbacks (GPL licence, not yet fully compatible under Windows) and the interface is not as clean as compared to lmfit, astropy. I made this experience for my specific optimisation problem, so please do not take this as given in general. ;-) With scipy.optimize.least_squares + TRF (default), I currently get better results in my case. So I was wondering if the (TRF) least_squares approach could be recommended in general over this transformation extension of LM?