ideas about stochastic minizers

[yorkerlin]

The stochastic minimizer framework (http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1FirstOrderSAGCostFunction.html http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1FirstOrderStochasticCostFunction.html http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1FirstOrderStochasticMinimizer.html) and the lossFunction framework for Vowpal Wabbit (http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CLossFunction.html) can be used to implement Shogun's online linear classifiers or the deep learning modular. (eg, http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html)

For example, we can extend the minimizer framework to support sparse vectors, GPU vectors and maybe parallel/asynchronous update.

BTW, the lossFunction is a subclass of SGObject. Data members in the lossFunction are not mutable variables. No idea why it is a subclass of SGObject? @iglesias I think all subclass of lossFunction are elements-wise cost functions.

My goal is to implement the deep GP or deep probabilistic models in the deep learning modular once my GP work is done.

[yorkerlin]

@karlnapf @lisitsyn @iglesias
any suggestion?

[yorkerlin]

ref https://github.com/shogun-toolbox/shogun/pull/2913

[yorkerlin]

I have implemented stochastic gradient descend (SGD), stochastic variance reduced gradient (SVRG), and stochastic mirror descend (SMD) minimizer. Next step is to implement stochastic coordinate descend (SCD).

For example:

• simple version : http://dl.acm.org/citation.cfm?id=1953048.2021059
• better version: http://arxiv.org/abs/1312.5799

For L1 (lasso) problem,

• SGD and SMD is good for the n > d case
• SCD is good for the d > n case

where n is the number of sample points, d is the number of features

[yorkerlin]

Next step:

• implement Coordinate descend methods for L1 penalty
• implement stochastic coordinate descend methods for for large scale L1 penalty

[karlnapf]

@yorkerlin hi, it might be a good idea to talk to John if you are doing VW related work. We always wanted to do another round of integrating Shogun and VW closer to each other. Send me an email if you want me to introduce you

[yorkerlin]

@karlnapf Yes. Please check out the email.