Using scipy's genetic algorithm for initial parameter estimation in gradient descent

[zunzun]

In your discussion of gradient descent, you point out the problem of initial parameter values for the gradient descent's starting point. For this reason, the authors of scipy added a genetic algorithm for initial parameter estimation. The module is named scipy.optimize.differential_evolution.

I have used scipy's Differential Evolution genetic algorithm to determine initial parameters for fitting a double Lorentzian peak equation to Raman spectroscopy data and found that the results were excellent. The GitHub project, with a test spectroscopy data file, is:

https://github.com/zunzun/RamanSpectroscopyFit

If you have any questions, please let me know. My background is in nuclear engineering and industrial radiation physics, and I love Python.

James Phillips

[NateZimmer]

Initial parameter estimation is a problem in all [NL]SQP optimization. My main problem with gradient descent is that its probably the most crappy optimization algorithm in existence short of brute force. The pecking order for optimization [ via gradient] is along the lines of Gradient-Descent < Gauss-Newton < Levenberg Marquadrt < LM + Trust region. There very few cases where Gradient Descent is an optimal algorithm to use. Granted, I did kinda take a dump on Gradient Descent before I documented the ones down the chain. Currently working on documenting those other algorithms.

In your case, looks like you used a global optimization algorithm that is more or less, heuristic brute force. Looks interesting and will certainly look into it. Sadly in my experience as a control systems engineer and with the problems I have to solve in a short time period(less than 10 seconds on cheap MCU), parameters will be so numerous that global estimation methods are not feasible.

[zunzun]

Scipy has several different non-linear solvers available for the scipy.optimize.curve_fit() routine, with the default solver being Levenberg-Marquardt. If no initial parameter values are passed to whichever non-linear solver is used, default starting values of all 1.0 are used - repeatedly causing the same types of problems for people who only know the characteristics of linear solvers.

The Differential Evolution genetic algorithm as implemented by scipy uses Latin Hypercube sampling of parameter space, requiring parameter bounds. This is why the Raman spectroscopy fitting code I linked to earlier uses parameter bounds. Note that for determination of initial parameters not all data need be used in every case, as a representative subset of data can be sufficient for this purpose.

I have some experience working with control systems on sheet steel, zinc coating, and paper product manufacturing so I understand your discussion, and although my area of expertise in these fields was measurement and related radiation physics I did rub elbows with the industrial control specialists and worked closely with them.

I assume that it is reasonable for you to now close this GitHub ticket.

[NateZimmer]

Yeah, scipy has a lot of amazing solvers. I think the Google Summer of Codes has contributed greatly to their wealth of diversity and quality. Though yeah, for a lot of problems, intelligent guesses need to be made or a system needs to be in place for intelligent guesses. Thanks for making the issue, I always enjoy discussion on the topics and I learned some new stuff looking at your repos so thanks ;) .