Using scipy's genetic algorithm for initial parameter estimation in gradient descent

[zunzun]

I see you are writing Python code for optimization on GitHub. A general problem for gradient descent and other non-linear algorithms - particularly for more complex equations - is the choice of initial parameters to start the "descent" in error space. Without good starting parameters, the algorithm will stop in a local error minimum. For this reason the authors of scipy have added a genetic algorithm for initial parameter estimation for use in gradient descent. The module is named scipy.optimize.differential_evolution.

I have used scipy's Differential Evolution genetic algorithm to determine initial parameters for fitting a double Lorentzian peak equation to Raman spectroscopy of carbon nanotubes and found that the results were excellent. The GitHub project, with a test spectroscopy data file, is:

https://github.com/zunzun/RamanSpectroscopyFit

If you have any questions, please let me know. My background is in nuclear engineering and industrial radiation physics, and I love Python, so I will be glad to help.

[tirthajyoti]

Thanks for bringing this to my attention.

So far, I have used Python library only for linear and logistic regression (binary classification) but I used the scikit-learn's default model fitting algorithm, so it is not in my hand.

The gradient descent code I wrote for linear regression is not in Python but Octave/MATLAB. I could definitely use a better initialization. However, because I used it for linear regression only where gradient descent with sufficiently small learning rate is almost guaranteed to find global minima, I did not pay too much attention to initialization.

I will focus on the initialization problem again when I use Python or MATLAB for back-propagation algorithm in multi-layer neural networks. I'll write to you later then.

[zunzun]

OK.