Comparison of GB & loss policies

arogozhnikov / hep_ml

Machine Learning for High Energy Physics.

https://arogozhnikov.github.io/hep_ml/

Other

179 stars 64 forks source link

Closed arogozhnikov closed 8 years ago

arogozhnikov commented 9 years ago

Probably some test notebook would be helpful to see how changes in losses affect training process.

arogozhnikov commented 9 years ago

Check on some weighted dataset several options:

sklearn (logloss, exploss)
xgboost (logloss)
UGBoost
- logloss, exploss only with target = gradient, weight = 1
- logloss, exploss with target = gradient/hessian, weight = hessian

arogozhnikov commented 9 years ago

todo: find out why there is such difference in variations of predictions

arogozhnikov commented 8 years ago

Some experiments I've done long ago say that:

UGradientBoosting update policy (all data + regularization) fails on the start, but turns out to be more stable when hundreds or thousands of trees are used.
see also discussion on kaggle, which confirms this point.

As of last point (setting target and weight), there seems to be no strong winner. Mostly following tradition, I'll prefer grad/hess version.