This Python package implements several boosting algorithms with different combinations of base learners, optimization algorithms, and loss functions.
Concerning base learners, KTBoost includes:
Concerning the optimization step for finding the boosting updates, the package supports:
The package implements the following loss functions:
It can be installed using
pip install -U KTBoostand then loaded using
import KTBoost.KTBoost as KTBoostFabio Sigrist
The package is build as an extension of the scikit-learn implementation of boosting algorithms and its workflow is very similar to that of scikit-learn.
The two main classes are KTBoost.BoostingClassifier and KTBoost.BoostingRegressor. The following code examples show how the package can be used. See also below for more information on the main parameters.
See also the Grabit demo for working examples of the Grabit model and the gamma regression demo for an example with the Gamma loss.
import KTBoost.KTBoost as KTBoost
################################################
## Define model (see below for more examples) ##
################################################
## Standard tree-boosting for regression with quadratic loss and hybrid gradient-Newton updates as in Friedman (2001)
model = KTBoost.BoostingRegressor(loss='ls')
##################
## Train models ##
##################
model.fit(Xtrain,ytrain)
######################
## Make predictions ##
######################
model.predict(Xpred)#############################
## More examples of models ##
#############################
## Boosted Tobit model, i.e. Grabit model (Sigrist and Hirnschall, 2017),
## with lower and upper limits at 0 and 100
model = KTBoost.BoostingRegressor(loss='tobit',yl=0,yu=100)
## KTBoost algorithm (combined kernel and tree boosting) for classification with Newton updates
model = KTBoost.BoostingClassifier(loss='deviance',base_learner='combined',
update_step='newton',theta=1)
## Gradient boosting for classification with trees as base learners
model = KTBoost.BoostingClassifier(loss='deviance',update_step='gradient')
## Newton boosting for classification model with trees as base learners
model = KTBoost.BoostingClassifier(loss='deviance',update_step='newton')
## Hybrid gradient-Newton boosting (Friedman, 2001) for classification with
## trees as base learners (this is the version that scikit-learn implements)
model = KTBoost.BoostingClassifier(loss='deviance',update_step='hybrid')
## Kernel boosting for regression with quadratic loss
model = KTBoost.BoostingRegressor(loss='ls',base_learner='kernel',theta=1)
## Kernel boosting with the Nystroem method and the range parameter theta chosen
## as the average distance to the 100-nearest neighbors (of the Nystroem samples)
model = KTBoost.BoostingRegressor(loss='ls',base_learner='kernel',nystroem=True,
n_components=1000,theta=None,n_neighbors=100)
## Regression model where both the mean and the standard deviation depend
## on the covariates / features
model = KTBoost.BoostingRegressor(loss='msr')#########################
## Feature importances ## (only defined for trees as base learners)
#########################
Xtrain=np.random.rand(1000,10)
ytrain=2*Xtrain[:,0]+2*Xtrain[:,1]+np.random.rand(1000)
model = KTBoost.BoostingRegressor()
model.fit(Xtrain,ytrain)
## Extract feature importances calculated as described in Friedman (2001)
feat_imp = model.feature_importances_
## Alternatively, plot feature importances directly
KTBoost.plot_feature_importances(model=model,feature_names=feature_names,maxFeat=10)
##############################
## Partial dependence plots ## (currently only implemented for trees as base learners)
##############################
from KTBoost.partial_dependence import plot_partial_dependence
import matplotlib.pyplot as plt
features = [0,1,2,3,4,5]
fig, axs = plot_partial_dependence(model,Xtrain,features,percentiles=(0,1),figsize=(8,6))
plt.subplots_adjust(top=0.9)
fig.suptitle('Partial dependence plots')
## Alternatively, get partial dependencies in numerical form
from KTBoost.partial_dependence import partial_dependence
kwargs = dict(X=Xtrain, percentiles=(0, 1))
partial_dependence(model,[0],**kwargs)In the following, we describe the most important parameters of the constructors of the two classes KTBoost.BoostingClassifier and KTBoost.BoostingRegressor.
loss : loss function to be optimized.
KTBoost.BoostingClassifier
{'deviance', 'exponential'}, optional (default='deviance')
'deviance' refers to the logistic regression loss for binary classification, and the cross-entropy loss with the softmax function for multiclass classification.
KTBoost.BoostingRegressor
{'ls', 'lad', 'huber', 'quantile', 'poisson', 'tweedie', 'gamma', 'tobit', 'msr'}, optional (default='ls')
'ls' refers to the squarred loss. 'lad' (least absolute deviation) is a robust version. 'huber' is a combination of the former two. 'quantile' does quantile regression (use 'alpha' to specify the quantile). 'tobit' corresponds to the Grabit model with a Tobit loss. 'msr' is a linear regression model where both the mean and the logarithm of the standard deviation are varying.
update_step : string, default="hybrid"
Defines how boosting updates are calculated. Use either "gradient" for gradient boosting or "newton" for Newton boosting (if applicable). "hybrid" uses a gradient step for finding the structur of trees and a Newton step for finding the leaf values. For kernel boosting, "hybrid" uses gradient descent. See the reference paper for more information.
base_learner : string, default="tree"
Base learners used in boosting updates. Choose among "tree" for trees, "kernel" for reproducing kernel Hilbert space (RKHS) regression functions, and "combined" for a combination of the two. See the reference paper for more information.
learning_rate : float, optional (default=0.1)
The learning rate shrinks the contribution of each base learner by 'learning_rate'. There is a trade-off between learning_rate and n_estimators.
n_estimators : int (default=100)
The number of boosting iterations to perform.
max_depth : integer, optional (default=5)
Maximum depth of the regression trees. The maximum depth limits the number of nodes in the tree. This value determines the interaction of the predictor variables.
min_samples_leaf : int, float, optional (default=1)
The minimum number of samples required to be at a leaf node:
min_samples_leaf as the minimum number.min_samples_leaf is a percentage and
ceil(min_samples_leaf * n_samples) are the minimum
number of samples for each node.min_weight_leaf : float, optional (default=1.)
The minimum number of weighted samples required to be at a leaf node. If Newton boosting is used, this corresponds to the equivalent (i.e., normalized) number of weighted samples where the weights are determined based on the second derivatives / Hessians.
criterion : string, optional (default="mse")
The function to measure the quality of a split. Supported criteria are "friedman_mse" for the mean squared error with improvement score by Friedman, "mse" for mean squared error, and "mae" for the mean absolute error.
random_state : int, RandomState instance or None, optional (default=None)
If int, random_state is the seed used by the random number generator;
If RandomState instance, random_state is the random number generator;
If None, the random number generator is the RandomState instance used
by np.random.
kernel : string, default="rbf"
Kernel function used for kernel boosting. Currently, supports "laplace", "rbf", and "GW" (generalied Wendland with "smoothness parameter" mu=1).
theta : float, default: 1.
Range parameter of the kernel functions which determines how fast the kernel function decays with distance.
n_neighbors : int, default: None
If the range parameter 'theta' is not given, it can be determined from the data using this parameter. The parameter 'theta' is chosen as the average distance of the 'n_neighbors' nearest neighbors distances. The parameter 'range_adjust' can be used to modify this. If range_adjust=3 or range_adjust=4.6, 'theta' is chosen such that the kernel function has decayed to essentially zero (0.05 or 0.01, respectively) at the average distance of the 'n_neighbors' nearest neighbors (for rbf and laplace kernel).
alphaReg : float, default: 1.
Regularization parameter for kernel Ridge regression boosting updates. This is added to the diagonal of the kernel matrix. Must be a non-negative number. A non-zero value helps to avoid singular matrices.
nystroem : boolean, default=None
Indicates whether Nystroem sampling is used or not for kernel boosting.
n_components : int, detault = 100
Number of data points used in Nystroem sampling for kernel boosting.
sigma : float, default=1 Standard deviation of the latent variable in a Tobit model.
yl : float, default=0 Lower limit of the Tobit model. If there is no lower censoring, simply set this parameter to a low value (lower than all data points).
yu : float, default=1 Upper limit of the Tobit model. If there is no upper censoring, simply set this parameter to a high value (higher than all data points).
gamma : float, default=1 Shape parameter for gamma regression.
tweedie_variance_power: float, default = 1.5 Tweedie power parameter for tweedie loss.