Optimization Techiniques for Machine Learning project work
Stochastic Gradient Descent with Momentum and Line Searches. Implementation, hyper-parameters tuning and efficiency testing of five variants of the Minibatch Gradient Descent algorithm on a benchmark of six real-world datasets, performing binary classification with Logistic Regression.
- SGD with fixed learning rate
- SGD with decreasing learning rate
- SGD with momentum (SGDM)
- SGD with Armijo line search for optimal learning rate during iterations
- SGD with momentum correction and restart, and Armijo line search (MSL-SGDM-C/R)