Method - update_terminal_regions in LossFunction Class- If Condition

[avinashbarnwal]

I think there might be some change required for update_terminal_regions method in LossFunction for If condition. We might need to change this to

if update_step == 'hybrid' or update_step == 'gradient': then line search of weights.

Please let me know if i am wrong.

[fabsig]

Thanks for your comment.

No, gradient boosting (update_step = 'gradient') does not update leave values. Only "hybrid gradient-Newton boosting" (update_step = 'hybrid') updates the leave values found with a gradient step by doing an additional Newton step. Note that the option update_step = 'hybrid' implements boosting as described in Friedman (2001). I denote this approach by "hybrid gradient-Newton boosting". See https://arxiv.org/abs/1808.03064 for further explanation in particular on the terminology "Newton boosting" and "hybrid gradient-Newton boosting".

Which loss function where you using? In the cases with loss functions where Newton updates are not applicable (e.g. 'lad', 'quantile' or 'huber') you need to use the option update_step = 'hybrid' to do the line search as in Friedman (2001).

[avinashbarnwal]

Thanks for the reply. I have gone through your paper thoroughly and understood ideas in the paper. I meant we need to add _update_terminal_region for classification loss function. I have added the code from the sklearn gradient boosting for classification loss function where we have _update_terminal_region for gradient boosting step. This might be not relevant for least square loss function as we pass on _update_terminal_region for gradient boosting.

https://gist.github.com/avinashbarnwal/29623bbe7f4b0aba22236659c03cb45c

[fabsig]

The "BinomialDeviance" loss already has an "_update_terminal_region" function, see https://github.com/fabsig/KTBoost/blob/master/KTBoost/KTBoost.py#L979, and also an "update_terminal_regions" function, which it inherits from its base class.

[avinashbarnwal]

Thanks.